Defining an interactive session that analyzes user input provided by a participant

ABSTRACT

In general, the subject matter described in this disclosure can be embodied in methods, systems, and program products for engaging in interactive sessions between computerized devices and participants. The method includes receiving user input that interacts with user controls to specify, for each respective content item in a plurality of content items, a text spelling and an audio recording. The method includes receiving user input that selects a set of user-selected content items from among the plurality of content items for inclusion as part of the first interactive exercise, and assigns an order to the user-selected content items. The computing system presents the selected content items in the user-selected order, receives user input that inputs the respective content items, and determines whether the user input matches the content item.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority to U.S. ProvisionalApplication No. 62/805,034, filed on Feb. 13, 2019, the disclosure ofwhich is herein incorporated by reference in its entirety.

TECHNICAL FIELD

This document relates to technologies that define an interactive sessionthat analyzes user input.

BACKGROUND

Various web sites and computer programs include content developed toeducate or improve skills of users of the respective web sites andcomputer programs. This content is typically not tailored to theindividual user, and the content takes a great amount of user effort togenerate. In other words, the process by which users interact with thecontent is not individualized and there is a significant burden on theauthor of the content to implement that content in acomputer-implemented environment.

SUMMARY

This document describes techniques, methods, systems, and othertechnologies for defining an interactive session that analyzes userinput. The following paragraphs list various implementations of thetechnologies described in this disclosure, with additional detailregarding those technologies provided in the figures and the detaileddescription section.

As additional description to the embodiments described below, thepresent disclosure describes the following embodiments.

First Set of Embodiments

Embodiment 1 is a computer-implemented method for defining and engagingin interactive sessions between computerized devices and participants.The method comprises performing, by a computing system, operations todefine at least a first interactive exercise, including by presenting,by the computing system, a first user interface that includes usercontrols configured to define a plurality of content items. Also byreceiving, by the computing system, user input that interacts with theuser controls of the first user interface to specify, for eachrespective content item in the plurality of content items,characteristics of the respective content item, including (a) a textspelling of the respective content item, and (b) an audio recording ofthe respective content item. Also by presenting, by the computingsystem, a second user interface that includes user controls configuredto define a plurality of interactive exercises that are each arranged topresent content items to users. Also by receiving, by the computingsystem, user input that interacts with the user controls of the seconduser interface to: (i) select a set of user-selected content items fromamong the plurality of content items for inclusion as part of the firstinteractive exercise, and (ii) assign an order to the user-selectedcontent items to produce an ordered set of user-selected content itemsthat are assigned to the first interactive exercise. The methodcomprises performing, by the computing system, operations to engage inat least the first interactive exercise, including by receiving, by thecomputing system, a request to engage a participant user in aninteractive exercise, and in response, selecting to engage theparticipant user in the first interactive exercise from among theplurality of interactive exercises. Also by accessing, by the computingsystem, the ordered set of user-selected content items based on theordered set of content items being assigned to the first interactiveexercise. Also by, for each respective content item, selected one at atime in order from among the ordered set of content items: (i)presenting, by the computing system, the respective content item, (ii)prompting, by the computing system, the participant user to interactwith the computing system to input the respective content item, (iii)receiving, by the computing system responsive to the computing systemprompting the participant user to interact with the computing system toinput the respective content item, participant user input, (iv)determining, by the computing system, whether the participant user inputmatches the respective content item, and (v) presenting, by thecomputing system, an indication regarding whether the computing systemdetermined the first participant user input to match the respectivecontent item.

Embodiment 2 the computer-implemented method of embodiment 1, whereineach content item in the plurality of content items comprises amulti-character word.

Embodiment 3 is the computer-implemented method of embodiment 1, whereinthe characteristics specified for each respective content item in theplurality of content items by the user input that interacts with theuser controls of the first user interface include (c) a user-specifiedphonetic representation of multiple component sounds of the respectivecontent item.

Embodiment 4 is the computer-implemented method of embodiment 3, whereinthe user-specified phonetic representation of the multiple componentsounds of the respective content item comprises a text string specifiedthrough user interaction with a keyboard.

Embodiment 5 is the computer-implemented method of any of embodiments3-4, wherein presenting the respective content item includes displayingthe text spelling of the respective content item that was specified bythe user input that interacted with the user controls of the first userinterface. Also receiving the participant user input includes recordinga participant audio recording using a microphone of the computingsystem. Also determining whether the participant user input matches therespective content item includes: (a) providing the participant audiorecording to a sound-identifying process that identifies componentsounds of the participant audio recording, and (b) determining whetherthe component sounds of the participant audio recording identified bythe sound-identifying process match the user-specified phoneticrepresentation of the multiple component sounds within the respectivecontent item.

Embodiment 6 is the computer-implemented method of any of embodiments3-5, wherein the user-specified phonetic representation of the multiplecomponent sounds of the respective content item identifies two or moresounds from among a collection of sounds, the collection of soundsincluding multiple sounds for each of various different alphabeticalletters.

Embodiment 7 is the computer-implemented method of embodiment 5, whereindetermining whether the component sounds of the participant audiorecording identified by the sound-identifying process match theuser-specified phonetic representation of the multiple component soundswithin the respective content item includes determining that a firstcomponent sound of the component sounds of the participant audiorecording matches a corresponding component sound of the multiplecomponent sounds within the respective content item. It also includesdetermining that a second component sound of the component sounds of theparticipant audio recording does not match a corresponding componentsound of the multiple component sounds within the first content item.Also, presenting the indication regarding whether the computing systemdetermined the first participant user input to match the respectivecontent item includes presenting (a) a text representation of therespective content item, (b) an indication that a first letter in therespective content item was spoken satisfactorily, the first lettercorresponding to the first component sound, and (c) an indication that asecond letter in the respective content item was spokenunsatisfactorily, the second letter corresponding to the secondcomponent sound.

Embodiment 8 is the computer-implemented method of embodiment 1, whereinreceiving the user input that interacts with the user controls of thesecond user interface assign the order to the user-selected contentitems to produce the ordered set of user-selected content items includesdisplaying, by the computing system, a collection of interface elementsthat correspond to the user-selected content items. Receiving, by thecomputing system, user input that interacts with a location at which oneinterface element from the collection of interface elements is displayedand drags the one interface element to a different location among thecollection of interface elements to change an order of the user-selectedcontent items.

Embodiment 9 is the computer-implemented method of embodiment 8, whereinreceiving the user input that interacts with the user controls of thesecond user interface to select the set of user-selected content itemsincludes displaying, by the computing system, the collection ofinterface elements that correspond to the user-selected content itemsselected for inclusion as part of the first interactive exercise and anadditional interface element that corresponds to an additional contentitem that is initially selected from among the plurality of contentitems for inclusion as part of the first interactive exercise. It alsoincludes receiving, by the computing system, user input that interactswith a location at which the additional interface item is displayed toremove the additional content item from the first interactive exerciseand leave the user-selected content items as those content itemsselected for the first interactive exercise.

Embodiment 10 is the computer-implemented method of embodiment 1,wherein presenting the respective content item includes outputting, withone or more speakers, the audio recording of the respective content itemthat was specified by the user input that interacted with the usercontrols of the first user interface. Receiving the participant userinput includes receiving user input that interacts with a touchscreen toform a freehand line drawing. Determining whether the participant userinput matches the respective content item includes: (a) identifying atext representation of the freehand line drawing, and (b) determiningwhether the text representation of the freehand line drawing matches thetext spelling of the respective content item specified by the user inputthat interacted with the user controls of the first user interface.Presenting the indication regarding whether the computing systemdetermined the first participant user input to match the respectivecontent item includes presenting (a) a text representation of therespective content item, and (b) an indication regarding whether thecomputing system determined the respective content item to be writtensatisfactorily, the computing system not presenting the textrepresentation of the respective content item while the computing systemwas receiving the user input that interacted with the touchscreen toform the freehand line drawing.

Embodiment 11 is directed to a computing system that comprises one ormore processors; and one or more computer-readable devices includinginstructions that, when executed by the one or more processors, causethe computing system to perform the method of any one of embodiments 1through 10.

Second Set of Embodiments

Embodiment 1 is a computer-implemented method for analyzing the mannerin which touchscreen user inputs are generated. The method comprisespresenting, by a computing system, a multi-character word and promptinga user of the computing system to write the multi-character word usingan input device of the computing system. The method comprises receiving,by the computing system, user input that interacts with the input deviceof the computing system to generate an user-animated gesture, responsiveto the computing system prompting the user to write the multi-characterword using the input device, wherein a final form of the user-animatedgesture satisfactorily represents the multi-character word. The methodcomprises identifying, by the computing system, different portions ofthe user-animated gesture that correspond to different respectivecharacters of the multi-character word. The method comprises, for eachrespective character in the multi-character word:

(i) accessing, by the computing system, a set of pre-stored animatedgestures that represent different techniques for forming the respectivecharacter; (ii) determining, by the computing system, a first level ofsimilarity between the portion of the user-animated gesture thatcorresponds to the respective character and a first pre-stored animatedgesture for forming the respective character from among the set ofpre-stored animated gestures for forming the respective character,wherein the first pre-stored animated gesture for forming the respectivecharacter is designated as representing a satisfactory technique inwhich to form the respective character; (iii) determining, by thecomputing system, a second level of similarity between the portion ofthe user-animated gesture that corresponds to the respective characterand a second pre-stored animated gesture for forming the respectivecharacter from among the set of pre-stored animated gestures for formingthe respective character, wherein the second pre-stored animated gesturefor forming the respective character is designated as representing anunsatisfactory technique in which to form the respective character; (iv)identifying, by the computing system, which of the first pre-storedanimated gesture and the second pre-stored animated gesture matches theportion of the user-animated gesture that corresponds to the respectivecharacter based on which of the first level of similarity and the secondlevel of similarity represents a greatest level of similarity; (v)storing, by the computing system, an indication regarding whether theuser-animated gesture formed the respective character using asatisfactory technique, based on which of the first pre-stored animatedgesture and the second pre-stored animated gesture the computing systemdetermined to match the user-animated gesture,

wherein the computing system identifies that the user-animated gestureformed a first character of the multi-character word using asatisfactory technique and identifies that the user-animated gestureformed a second character of the multi-character word using anunsatisfactory technique; and

displaying, by the computing system, an indication that theuser-animated gesture formed the second character using anunsatisfactory technique, based on the computing system determining thata portion of the user-animated gesture that corresponds to the secondcharacter has a greatest level of similarity with a pre-stored animatedgesture that represents an unsatisfactory technique for forming thesecond character rather than a pre-stored animated gesture thatrepresents a satisfactory technique for forming the second character.

Embodiment 2 is the computer-implemented method of embodiment 1, whereinpresenting the multi-character word includes displaying a textualrepresentation of the multi-character word on a touchscreen.

Embodiment 3 is the computer-implemented method of embodiment 1, whereinidentifying which of the first pre-stored animated gesture and thesecond pre-stored animated gesture matches the user-animated gesturebased on which of the first level of similarity and the second level ofsimilarity represents a greatest level of similarity includesdetermining that one of the first level of similarity and the secondlevel of similarity satisfies a designated threshold level ofsimilarity.

Embodiment 4 is the computer-implemented method of embodiment 1, whereinthe first pre-stored animated gesture for forming the respectivecharacter represents a technique in which the respective character isformed by starting at a first location of the respective character. Thesecond pre-stored animated gesture for forming the respective characterrepresents a technique in which the respective character is formed bystarting formation of the respective character starting at a secondlocation of the respective character, the second location being on adifferent side of the respective character than the first location.

Embodiment 5 is the computer-implemented method of embodiment 1, whereinthe method further comprises, for each respective character in themulti-character word, determining, by the computing system, a thirdlevel of similarity between the portion of the user-animated gesturethat corresponds to the respective character and a third pre-storedanimated gesture for forming the respective character from among the setof pre-stored animated gestures for forming the respective character.The second pre-stored animated gesture for forming the respectivecharacter represents a first type of unsatisfactory technique in whichto form the respective character. The third pre-stored animated gesturefor forming the respective character represents a second type ofunsatisfactory technique in which to form the respective character. Thecomputing system identifying that the user-animated gesture formed thesecond character of the multi-character word using an unsatisfactorytechnique includes the computing system identifying that theuser-animated gesture formed the second character using the first typeof unsatisfactory technique. The computing system is configured toidentify that the user-animated gesture formed the second characterusing the second type of unsatisfactory technique responsive to thecomputing system determining that the portion of the user-animatedgesture that corresponds to the second character has a greatest level ofsimilarity with the third type of pre-stored animated gesture accessedfor the second character.

Embodiment 6 is the computer-implemented method of embodiment 5, whereinthe second type of unsatisfactory technique represented by the thirdpre-stored animated gesture for forming the respective characterincludes the respective character being formed backwards.

Embodiment 7 is the computer-implemented method of embodiment 1, whereindetermining, by the computing system, the first level of similarityincludes coordinating, by the computing system, a timing of the portionof the user-animated gesture that corresponds to the respectivecharacter and a first timing of the first pre-stored animated gesturefor forming the respective character. The determining also includesdetermining, for each respective time of multiple first coordinatedtimes after the timing and the first timing have been coordinated, afirst distance between (a) a location of user input forming theuser-animated gesture at the respective time, and (b) a locationrepresenting user input forming the first pre-stored animated gesture atthe respective time. The determining also includes generating, by thecomputing system, the first level of similarity based on a combinationof the first distances determined for the multiple first coordinatedtimes. Determining, by the computing system, the second level ofsimilarity includes coordinating, by the computing system, the timing ofthe portion of the user-animated gesture that corresponds to therespective character and a second timing of the second pre-storedanimated gesture for forming the respective character. The determiningalso includes determining, for each respective time of multiple secondcoordinated times after the timing and the second timing have beencoordinated, a second distance between (a) a location of user inputforming the user-animated gesture at the respective time, and (b) alocation representing user input forming the second pre-stored animatedgesture at the respective time. The determining also includesgenerating, by the computing system, the second level of similaritybased on a combination of the second distances determined for themultiple second coordinated times.

Embodiment 8 is the computer-implemented method of embodiment 1, furthercomprising presenting, by the computing system, a second multi-characterword and prompting the user of the computing system to write the secondmulti-character word using the input device of the computing system. Themethod includes receiving, by the computing system, user input thatinteracts with the input device to generate a second user-animatedgesture, responsive to the computing system prompting the user to writethe second multi-character word with the input device. The methodincludes identifying, by the computing system, different portions of thesecond user-animated gesture that correspond to different respectivecharacters of the second multi-character word. The method includesdetermining, by the computing system through a comparison of the seconduser-animated gesture to the second multi-character word, that thesecond user-animated gesture does not match the second multi-characterword, and instead matches the second multi-character word with one of(a) an extra letter inserted, (b) a letter missing, and (c) lettersordered incorrectly. The method includes displaying, by the computingsystem, an indication that the second user-animated gesture formed thesecond multi-character word with one of (a) an extra letter inserted,(b) a letter missing, and (c) letters ordered incorrectly, based on thecomparison of the second user-animated gesture to the secondmulti-character word.

Embodiment 9 is the computer-implemented method of embodiment 1, whereinthe computing system includes a server system, and the server systemperforms (i) the determining the first level of similarity between theportion of the user-animated gesture and the first pre-stored animatedgesture, and (ii) the determining the second level of similarity betweenthe portion of the user-animated gesture and the second pre-storedanimated gesture. The presenting the multi-character word involves theserver system providing information to a handheld computing device tocause the handheld computing device to present the multi-character word.The receiving the user input involves the server system receiving fromthe handheld computing device an indication that the user input wasprovided at the handheld computing device. The displaying the indicationthat the user-animated gesture formed the second character using anunsatisfactory technique involves the server system providinginformation to a second computing device to cause the second computingdevice to display text that specifies that the user of the computingsystem formed the second character using an unsatisfactory technique.

Embodiment 10 is directed to a computing system that comprises one ormore processors; and one or more computer-readable devices includinginstructions that, when executed by the one or more processors, causethe computing system to perform the method of any one of embodiments 1through 9.

Third Set of Embodiments

Embodiment 1 is a computer-implemented method of interaction between acomputer and user in forming statements, the method comprisespresenting, by a computing system through a user interface, a firstmulti-character word. The method comprises prompting, by the computingsystem, a user to interact with the computing system to input the firstmulti-character word. The method comprises receiving, by the computingsystem responsive to the computing system prompting the user to interactwith the computing system to input the first multi-character word, firstuser input. The method comprises determining, by the computing system,that the first user input does not match the first multi-character word.The method comprises performing, by the computing system after havingdetermined that the first user input does not match the firstmulti-character word, operations to present the user with a similar orsame, second multi-character word, including by presenting, by thecomputing system, a text representation of the second multi-characterword on a display device, and also by, for each respective character ofthe second multi-character word, in order one at a time: (i) monitoring,by the computing system, an input device for user input that contacts alocation at which the display device presents the respective character,(ii) determining, by the computing system, that user input has used theinput device to contact the location at which the display devicepresents the respective character, and (iii) displaying, by thecomputing system, an indication on the display device that user inputhas contacted the respective character. The method comprises presenting,by the computing system, an indication that user input has completedcontacting all character in the second multi-character word.

Embodiment 2 is the computer-implemented method of embodiment 1, whereinpresenting the first multi-character word includes outputting an audiblerepresentation of the first multi-character word using one or morespeakers. Receiving the first user input includes receiving user inputthat interacts with a touchscreen to trace a gesture.

Embodiment 3 is the computer-implemented method of embodiment 1, whereindetermining that the first user input does not match the firstmulti-character word includes analyzing the first user input anddetermining that the first user input satisfactorily represents multiplecharacters of the multi-character word and that the first user inputunsatisfactorily represents a first character of the multi-characterword. The method further comprises identifying, by the computing system,the second multi-character word, including by accessing, by thecomputing system, a plurality of candidate words. Also by parsing, bythe computing system, each of the plurality of candidate words toidentify a selected word that: (i) is a same length as the firstmulti-character word, (ii) includes the multiple characters that thefirst user input satisfactorily represented in same locations within theselected word as the multiple characters are located in the firstmulti-character word, and (iii) includes a different character that isdifferent than the first character that the first user inputunsatisfactorily represented in a same location within the selected wordas the first character is located within the first multi-character word,wherein the second multi-character word is same as the selected word andis different from the first multi-character word.

Embodiment 4 is the computer-implemented method of embodiment 1, whereindetermining that user input has used the input device to contact thelocation at which the display device presents the respective characterincludes determining that user input has tapped the location at whichthe display device presents the respective character.

Embodiment 5 is the computer-implemented method of embodiment 1, furthercomprising for each respective character of the second multi-characterword, in order one at a time displaying, by the computing system, aninterface element marker at a location of the respective character atwhich user input is to contact the respective character. Also one at atime determining, by the computing system that user input has contactedthe location at which the display device presented the interface elementmarker. Also one at a time receiving, by the computing system, anindication that user input has moved from the location at which thedisplay device presented the interface element marker across the displaydevice in a path that matches a displayed representation of therespective character. Also one at a time moving, by the computingsystem, a display of the interface element marker along the displayedrepresentation of the respective character as the user input movesacross the display device, wherein the computing system is configured tonot move the interface element marker away from the representation ofthe respective character in response to user input moving away from therepresentation of the respective character. Also one at a timedetermining, by the computing system, that the user input has traced anentirety of the displayed representation of the respective character,and in response, displaying an indication that the respective characterhas been completely traced and designating a next character of thesecond multi-character word for tracing if the respective character isnot a last character of the second multi-character word.

Embodiment 6 is the computer-implemented method of embodiment 5, furthercomprising, displaying, by the computing system a traced portion of therespective character over which the interface element marker has passedin a different manner than an untraced portion of the respectivecharacter over which the interface element marker has not passed.

Embodiment 7 is the computer-implemented method of any of embodiments1-5, further comprising determining, by the computing system, that userinput has contacted each character of the second multi-character word.The method also includes displaying, by the computing system after thecomputing system has determined that user input has contacted eachcharacter of the second multi-character word, a second interface elementmarker at a first location of the display device that is closer on thedisplay device to the first character of the second multi-character wordthan any other letter of the second multi-character word. The methodalso includes determining, by the computing system, that user input hascontacted the location at which the second interface element isdisplayed and has moved across the display device towards other lettersin the second multi-character word. The method further comprises moving,by the computing system, a display of the second user interface markeracross the display device to correspond to a location of user inputcontacting the display device, wherein the computing system isconfigured to not move the second user interface marker across thedisplay device responsive to user input contacting a location at whichthe second interface element marker is not displayed. The method alsoincludes changing, by the computing system, a displayed representationof each character in the second multi-character word on the displaydevice one at a time as user input moves the second interface elementmarker past the respective character to indicate that the respectivecharacter has been selected, until user input has moved the secondinterface element to a last character in the second multi-characterword.

Embodiment 8 is the computer-implemented method of embodiment 7, furthercomprising outputting, by the computing system, an audio recording ofthe second multi-character word with one or more speakers while userinput moves the second interface element marker across the displaydevice.

Embodiment 9 is the computer-implemented method of embodiment 8, furthercomprising identifying portions of the audio recording of the secondmulti-character word that represent respective characters within themulti-character word. The outputting of the audio recording of thesecond multi-character word includes audibly outputting each portion ofthe audio recording of the second multi-character word in coordinationwith user input moving the second interface element marker across thedisplay device, such that movement of the second interface elementmarker controls timing of the audible output of each portion of theaudio recording of the second multi-character word.

Embodiment 10 is the computer-implemented method of embodiment 1,wherein the first multi-character word is part of an ordered collectionof multi-character words that the computing system presents one at atime as part of an interactive session between the computing system andthe user. The computing system is configured, responsive to thecomputing system determining that the first user matches the firstmulti-character word, to skip performance of the operations to presentthe user with the similar or same, second multi-character word, andinstead: (i) present a next multi-character word from the orderedcollection of multi-character words through the user interface, the nextmulti-character word including multiple characters that are differentfrom the first multi-character word, (ii) prompt the user to interactwith the computing system to input the first multi-character word, (iii)receive next user input, (iv) determine whether the next user inputmatches the next multi-character word.

Embodiment 11 is directed to a computing system that comprises one ormore processors; and one or more computer-readable devices includinginstructions that, when executed by the one or more processors, causethe computing system to perform the method of any one of embodiments 1through 10.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features,objects, and advantages will be apparent from the description anddrawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 shows a computing system for defining an interactive session thatanalyzes user input and engaging in the interactive session.

FIG. 2 shows a flowchart of a process for specifying a collection ofcontent items for interactive sessions.

FIGS. 3A-B show user interfaces for defining content items.

FIG. 4 shows a flowchart of a process for defining an interactivesession.

FIGS. 5A-B show user interfaces for defining interactive sessions andinteractive exercises.

FIG. 6A-B show a flowchart of a process for engaging in an interactivesession.

FIG. 7 shows a flowchart of a process for performing a first interactiveexercise.

FIGS. 8A-D show user interfaces for performing a first interactiveexercise.

FIG. 9 shows a flowchart of a process for determining whether recordedaudio matches a content item.

FIG. 10 shows a flowchart of a process for a completion routine.

FIGS. 11A-B show user interfaces of the completion routine.

FIG. 12 shows a flowchart of a process for performing a secondinteractive exercise.

FIGS. 13A-C show user interfaces for performing the second interactiveexercise.

FIG. 14 shows a flowchart of a process for performing a thirdinteractive exercise.

FIGS. 15A-B show user interfaces for performing the third interactiveexercise.

FIGS. 16A-B show a flowchart of a process for performing a firstcorrection routine.

FIG. 17A-D show user interfaces for performing the first correctionroutine.

FIGS. 18A-B show a flowchart of a process for performing a fourthinteractive exercise.

FIG. 19A-B show user interfaces for performing the fourth interactiveexercise.

FIGS. 20A-B show a flowchart of a process for performing a secondcorrection routine.

FIGS. 21A-F show user interfaces for performing the second correctionroutine.

FIG. 22 shows a flowchart of a gesture-comparison process.

FIGS. 23A-B show a flowchart of a process for performing a fifthinteractive exercise.

FIGS. 24A-B show user interfaces for performing the fifth interactiveexercise.

FIG. 25 shows a flowchart of a process for performing a sixthinteractive exercise.

FIG. 26 shows a flowchart of a process for performing a seventhinteractive exercise.

FIGS. 27A-D show user interfaces for performing the seventh interactiveexercise.

FIGS. 28A-B show a flowchart for analyzing user interaction during theinteractive exercises.

FIGS. 29A and 29C show user interfaces for analyzing user interactionduring the interactive exercises.

FIG. 29B shows cards that identify content items that a participantincorrectly input.

FIG. 30A shows a user interface for providing a participant with avirtual gift.

FIG. 30B shows a user interface with which a participant can customize acharacter.

FIG. 31 is a conceptual diagram of a system that may be used toimplement the systems and methods described in this document.

FIG. 32 is a block diagram of computing devices that may be used toimplement the systems and methods described in this document, as eithera client or as a server or plurality of servers.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

This document describes technologies to define an interactive exercisethat analyzes user input. The interactive sessions described in thisdisclosure involve a computing system outputting information to aparticipant user, and the computing system prompting the participant torespond by supplying input back to the computing system. In someexamples, the information is audibly output (e.g., one or more speakersmay play a recording of a letter, a word, or a sentence). In response,the computing system can display an input field on a touchscreen deviceinto which the participant can write a graphical representation of theaudible output, for example, with a stylus or finger. In other examples,the information is graphically output on a display device and theparticipant is prompted to repeat the information back to the computingsystem by speaking the information into a microphone of the computingsystem. The computing system can record the participant's spokenresponse.

The computing system analyzes the participant's written or spokenresponse to determine whether the response matches the system output. Inthose situations in which the computing system output the informationgraphically and the participant responded by providing a spokenresponse, the computing system may separate the recording of theparticipant-spoken response into its component sounds and may thencompare those component sounds to a list of component sounds thatrepresent the system output, as previously specified by an administratorthat during a process that defined various different characteristics ofthe content output by the computing system.

In those situations in which the computing system output the informationaudibly and the participant responded with a written response, thecomputing system may initially compare the written response to thesystem output to determine whether the participant's written responsematches the system output. The computing system can further process theparticipant's written response by separating an animated gesture thatforms the participant's written response into component animatedgestures that represent each component character of a word representedby the participant-written response. The computing system can thencompare each component animated gesture to a collection of pre-storedanimated gestures for each respective character of the word or sentenceaudibly output by the computing system to determine which pre-storedanimated gesture is a best match. The identification of thebest-matching pre-stored animated gesture enables the computing systemto determine a technique in which the participant wrote each respectivecharacter. The system can indicate to the participant or another user ofthe computing system the techniques that the participant uses to writevarious characters and whether those techniques represent satisfactorytechniques for forming various characters.

Each portion of information that the computing system outputs to aparticipant is termed a content items in this disclosure, and thecollection of content items that form the basis for interactive sessionsbetween the computing system and participants is specified by a userwith authorization to define interactive sessions (e.g., anadministrative user). Such a user may select not only which contentitems form the basis of each interactive session, but also may define anorder in which the content items are to be provided to participantsduring the interactive sessions, and the order in which the computingsystem is to provide multiple interactive sessions to participants.

Prior to a user with appropriate authorization creating an interactivesession, that user can define the characteristics of various contentitems in the collection of content items from which users can populateinteractive sessions. For example, a user adding a new content item tothe collection, such as the word “house,” can specify (i) a textrepresentation of the new content item, (ii) an audio recording thatrepresents the new content item, and (iii) a phonetic representation ofthe new content item, for example, a text string that defines each soundwithin the new content item.

FIG. 1 shows a computing system for defining an interactive session thatanalyzes user input and engaging in the interactive session. Thecomputing system includes various different computing devices, includingdevices 110, 130, 150, and 160. Devices 150 represent tablet computingdevices used by participants to engage in interactive sessions where thecomputing devices display or audibly output content items, andparticipants are prompted to respond by speaking or writing the contentitems back to respective computing devices 150.

Device 110 represents a laptop computing device at which a user is ableto define characteristics of content items, create and edit interactivesessions, and/or view metrics of participants that have engaged in theinteractive sessions. Device 160 represents a tablet computing device160 at which an individual that is instructing participants can accessthe computing system to create accounts for participants, identifyprogress of participants, and view metrics for participants that areassigned to the that individual instructor.

Devices 110, 150, and 160 communicate over a network 140 (e.g., theInternet) with a server system 130 that interacts with the devices, forexample, under a client-server model or providing responses to queriessent by the devices. For instance, devices 110 and 160 may presentcontent in web pages that are hosted by the server system 130, anddevices 150 may include application programs that engage users of thedevices 150 in the interactive sessions and send queries to the serversystem 130 for content, with the server system 130 responding to thequeries by providing the content back to the devices 150 forpresentation by the application programs.

Although the above description indicates that devices 110 and 160present content related to the interactive sessions in web pages, anddevices 150 present content related to the interactive sessions inapplication programs, any of the devices may present such content in webpages and any of the devices may present such content in applicationprograms. Server system 130 may access the content and metrics datastore 120 to select information to send to the devices 110, 150, and160. The contents and metrics data store 120 represents one or moredifferent mechanisms for storing information (e.g., one or moredatabases or data files).

Devices 110, 130, 150, and 160 work together as a computing system, andvarious references in this disclosure to a computing system performingvarious operations can involve various different levels of interactionbetween the devices. As an example, reference to the computing systempresenting content can involve a participant device 150 presentingcontent on a display of the device 150 or the server system 130providing information for receipt by the participant device 150 to causethe participant device to present the content on the display of thedevice 150. Similarly, reference to the computing system determiningwhether user input matches content previously output by the computingsystem can include the participant device 150 performing the matchingdetermination itself or requesting that the server system 130 to performthe matching determination and receiving back from the server system 130an indication regarding whether there the server system 130 determinedthere to be a match. As such, the amount of processing performed locallyon devices 110, 150, and 160 (and conversely the amount of processingperformed remote at the server system 130) varies depending on theimplementation.

Moreover, although the device 110 is illustrated as a laptop and devices150 and 160 are illustrated as tablet computers, each of these devicescan be other types of computing devices, such as smartphone computingdevices, tablet computing devices, laptop computing devices, desktopcomputing devices, or similar types of portable or non-portablecomputing devices that include components to output content to users andreceive user input in response.

The computing system described in this disclosure provides variousbenefits, including allowing an author of content to define interactivesessions in an efficient manner and to offload the responsibility forproviding content that forms the basis of those interactive sessions tosemi-automated computing processes. The efficiencies borne by the systemare partially due to the ability of the administrator to define alibrary of content items that are re-usable among different types ofinteractive sessions.

Additional details of the computing system discussed in this disclosureare provided with reference to FIGS. 2 through 32 , which includeflowcharts that identify operations of the computing system andscreenshots that illustrate example user interfaces output by devices110, 150, and 160.

Definition of Content Items and Interactive Sessions

FIG. 2 shows a flowchart of a process for specifying a collection ofcontent items for interactive sessions. This process may be performedthrough interaction between the computing system described herein and aperson with administrative and/or instruction credentials once thatperson has logged into the computing system.

As background before describing details of the process for specifyingthe collection of content items, a person with administrativecredentials may sit down at the computer 110 (FIG. 1 ), use a webbrowser to navigate to a web page administered by the server system 130,and enter a username and password. The server system 130 mayauthenticate the user, determine that the user has an administrator orinstructor role, and provide the user interface that is shown in FIG.3A. The FIG. 3A user interface includes a left pane 310 that includesvarious interface element tabs, and FIG. 3A shows that the user hasselected the “Manage Sounds” tab. Accordingly, the main portion 320 ofthe user interface that is shown to the right of the left pane 310includes interface elements with which the user can manage content items(which various user interfaces shown in the figures refer to as“sounds”), including a “Missing Content/Chunk” interface element 322, a“Review Sound Library” interface element 324, and an “Add New/UpdateContent” user interface element 326.

The “Add New/Update Content” interface element is selected in FIG. 3A,and it is through interaction with fields 330 through 340 with whichuser input is able to specify characteristics of a new content item.User selection of the “Review Sound Library” interface element 324causes the computing system to switch from the FIG. 3A user interface tothe FIG. 3B user interface. The user interfaces of FIG. 3A and FIG. 3Bare similar, except that the FIG. 3A user interface allows a user tocreate a new content item, while the FIG. 3B user interface allows auser to view a list of existing content items and edit thecharacteristics for those content items.

Referring now to the flowchart of FIG. 2 , at box 210 the computingsystem defines a content item. As mentioned above, a content item may bedefined either through the user interface of FIG. 3A (creating a newcontent item) or through the user interface of FIG. 3B (editing anexisting content item).

At box 212, the computing system receives user input that specifies textfor a content item. For example, a user may select input field 330 (FIG.3A) and type the letters of the word “tap” using a virtual or physicalkeyboard. As another example, a user may select the cell in the firstrow of column 352 (FIG. 3B) to edit the text for the correspondingcontent item to read “tap” (assuming that the text was previouslydifferent). These user inputs illustrate various ways to define theuser-specified text for a content item.

At box 214, the computing system receives user input that specifiesaudio for the content item. For example, through interaction withinterface elements 332, 334, and 336, a user may upload a soundrecording for a content item. Element 332 may include controls to recordaudio, so that a user record audio that corresponds to the text “tap”without using a separate audio-recording application program. Thecomputing device at which the user is sitting would use a microphone torecord the audio and would store a digital representation of an audiowaveform as an audio file.

The controls may also include an option to upload a previously-recordedaudio file that includes audio that corresponds to the text “tap.” Theuser can interact with interface element 334 to preview the soundrecording, and can select interface element 336 to download the soundrecording. Similar functionality is provided in the “editing” userinterface of FIG. 3B. In particular, the a user may select controls inthe first row in column 354 to play, download, or change the audiorecording for the “tap” content item.

At box 216, the computing system receives user input specifyingcomponent sounds for the content item. The “Add New” user interface(FIG. 3A) includes a “Sound Chunk” field 338 for specifying thisinformation. A similar field in the “Edit” user interface (FIG. 3B) isrepresented by the entries under the “Chunk” column 356. The Chunk entryfor the “tap” interface element is the string “t,a_short,p” in FIG. 3B.This text string identifies the three phonetic components for thecontent item “tap”, namely: (1) “t”, (2) short “a”, and (3) “p.”

The computing system may store, in data store 120, information for eachof the forty-four sounds in English including for each sound: (i) dataidentifying how to identify the respective sound within user-generatedaudio, and (ii) a text representation of the sound. FIG. 3B shows thatthe text representation of the sound “t” is simply “t”, that the textrepresentation of the sound of a short “a” is “a_short”, and that thetext representation for the sound “p” is also simply “p”.

Fields 338 (FIG. 3A) and 356 (FIG. 3B) allow a user to specify thecomponent sounds that as a whole comprise a content item, in thisexample as a text string with commas separating each portion of textthat represents each respective sound (box 217). The string in the FIG.3B example does not include spaces, but in other examples spaces may bepermitted. Also, the computing system could present a user with a listof component sounds and the user may select those sounds from the listthat represent how the content item is stated. The user selection of thesounds may involve clicking with a mouse or tapping with a finger thoseinterface elements that represent the sounds in an order that representsthe spoken output of the respective content item.

At box 218, the computing system stores the characteristics defined forthe content item data store 120. As described above, the storedinformation may include at least (1) a text representation of thecontent item, (2) an audio recording for the content item, and (3)information that identifies the various sounds that as a whole comprisethe content item, and an order in which those sounds are audibly outputto represent the content item. Although the number sounds that comprisethe content item can be the same as the number of letters (e.g., threesounds for three letters, as with the word “tap”), the number of soundsand letters can be different. The computing system may access theinformation stored for a content item when a user defines an interactivesession, and when the computing system engages in an interactive sessionthat presents the content item, as described in more detail below.

FIGS. 3A and 3B include additional interface elements not yet mentioned.The interface elements in the table cells under column 350 show theinteractive sessions (termed “lessons” in the figures) into which thecontent item has been added. The interface elements in the table cellsunder column 358 allow a user to delete respective content items fromthe collection of content items. At the top of columns 350, 352, 354,356, and 358 are filter fields that allow users to input text to filterthe list of content items to a subset of content items that match theinput text that. And the “Missing Content/Chunk” interface element 322shows the content items that are missing both audio and “chunk”information.

FIG. 4 shows a flowchart discussing processes related to the “ManageLessons” user interfaces of FIGS. 5A-B. Through interaction with theseuser interfaces, an individual that has logged into the computer systemusing administrative and/or instruction credentials can define aninteractive session, as the flowchart of FIG. 4 describes.

At box 410, a computing system creates, edits, moves, and/or archives aninteractive session responsive to user input. An example interface forperforming such actions is shown in FIG. 5A, which shows a table 502that includes ten rows that include interface elements for tenrespective interactive sessions. These are ten interactive sessions thatthe user or other users with administrative and/or instructorcredentials have already created and, as discussed in detail later, thecomputing system is designed to provide the interactive sessions toparticipant users in order, for example, so that after a participantuser has participated in “Doodle Lesson 002” and finished that lesson,the computing system would then engage the participant user in “DoodleLesson 003.”

The FIG. 5A user interface includes various different interface elementcontrols that enable users to interact with and edit the interactivesessions. For example, a user can select a clone interface element 512to create a copy of an interactive session. A user can select an upinterface element 516 to move the corresponding interactive session up,and can select a down interface element 518 to move the correspondinginteractive session down. A user can select an archive interface element520 to move the interactive session to a different user interface (e.g.,as an alternative to permanently deleting the interactive session).Particularly relevant to the discussion below, a user may select acreate lesson interface element 510 to create a new interactive session,or may select an edit interface element 514 to edit an already-createdinteractive session.

Selection of the edit interface element 514 causes the computing systemto present the user interface that is shown in FIG. 5B. The FIG. 5B userinterface includes a table 550 that includes seven rows 520 a-g forseven respective interactive exercises that together comprise theinteractive session. The table 550 includes eight columns 530 through542 that each include fields and controls to define characteristics ofthe interactive exercises.

For example, the first row 520 a in table 550 is for the interactiveexercise “Flash Doodle”, as specified by the words “Flash Doodle” incolumn 530 (labelled the “Activity” column). At Box 420, a user caninteract with the controls in the cells of row 520 a to define thecontent and operation of the “Flash Doodle” interactive exercise. Aswill be illustrated in detail below, an interactive exercise involves acomputing system interacting with a participant to perform a series oftasks based on content items selected for the interactive exercise by anadministrative user. The content items that form the basis for theseparticipant-performed tasks are defined in the cells of the “gamecontent” column 542.

At box 422, an administrative user adds content items to the FlashDoodle interactive exercise by interacting with the game content cell542 a. In this example, a user selects blank space within the gamecontent cell 542 a, for example, with a mouse click or finger tap.Responsive to the user selection of the blank space, the computingsystem presents a dialog box (not shown) that includes a list ofcandidate content items. The list of candidate content items may includeall content items from the library of content items (e.g., all items inthe “sound library” shown in FIG. 3B). User selection of a content itemfrom the list may cause the computing system to add the selected contentitem to the list of content items shown in the game content cell 542 a.The selected content item may remain available for inclusion in otherinteractive exercises.

The game content cell 542 a shows a list of a couple dozen differentcontent items. At box 424, the computing system may receive user inputthat reorders or removes content items. For example, the computingsystem may receive user input that selects the interface element for oneof the content items in the game content cell 542 a and drags theselected interface element to a different location within the gamecontent cell 542 a, changing the order of the interface elements andtherefore also the order in which the computing system will present thecontent items during the corresponding interactive exercise. Thecomputing system may also receive user input that selects the “x” markat the right side of the interface element for each content item toremove the respective content item from the corresponding interactiveexercise. The order and selection of content items represented withinthe game content cell 542 a is important because the computing systemmay present the content items in the order specified in the game contentcell 542 a during an actual performance of the interactive exercise.

At box 426, the computing system receives user input that specifies thetime, score, and other criteria for an interactive exercise. Forexample, a user may change the maximum time value (in minutes) in a cellunder the “time” column 532, to change the maximum time (in minutes) foran interactive exercise before the computing system moves to the nextinteractive exercise. As previously described, each interactive exercisemay involve presenting a collection of content items one at a time to aparticipant user and asking the participant user to respond to thepresentation of each content item by either saying the content item orwriting the content item. Upon expiration of a timer, the system mayeither immediately end the interactive exercise, or permit theparticipant user to complete the task for the currently-presentedcontent item before ending the interactive exercise.

As another example, the cells under the “banana score” column 534 allowa user to specify the score that a participant user can earn whencompleting a certain number of tasks correctly. The cells in the“correct set” column 536 allow a user to specify the number of tasksthat a participant must perform correctly in order to earn the scorethat is specified in the banana score column. The cells in the “gamecompletion score” column 538 allow an administrative user to specify thescore that a participant user earns upon completing the correspondinginteractive exercise. The check boxes in the “select for proficiency”column 542 allow an administrative user to select whether or not metricsfor the respective interactive exercise are provided for review by aninstructor user.

The discussion provided above explains how an administrative user maydefine the characteristics of a single interactive exercise—the FlashDoodle interactive exercise, corresponding to row 520 a in thisexample—and an administrative user may repeat this process for each ofthe other six interactive exercises illustrated in table 550 in FIG. 5B(as indicated by the “More exercises” decision box 430 in the FIG. 4flowchart). An exception is that the “Doodle House” row 520 g includesthree sub-rows for “Phonemes,” “Read Words,” and “Write Words.” This isbecause the Doodle House interactive exercise is actually a combinationof different exercises that read and write phonemes and words. As such,the administrator is able to separately specify which phonemes areinvolved in that interactive exercise, which words are to be read, whichwords are to be written, and the scoring and various other metrics foreach sub-portion of the Doodle House interactive exercise.

At box 440, the computing system receives user input that specifiesre-direct content items. The re-direct content items are represented inFIG. 5B in field 560. These are content items that are used for thefirst and second correction routines that are described with respect toFIGS. 16 a-b and 20 a-b . An administrative user can select and orderthese re-direct content items in the same manner as described withrespect to the cells in the game content column 542.

The fields in the reward criteria table 570 specify the score that thecomputing system awards a participant user upon completing variousinteractive exercises if criteria are satisfied. The values specified intable 570 show that the computing system awards a participant user asingle banana if that participant completes the Flash Doodle interactiveexercise with 25% of the tasks correctly performed, and two bananas ifthe participant user completes the same interactive exercise with 50% ofthe tasks correctly performed. Similarly, if the user completes the GoDoodle interactive exercise while providing the correct capitalizationfor every content item (e.g., sentences), the computing system awardsthe user a single banana.

Participation in Interactive Sessions

FIG. 6A shows a flowchart of the overall process for a computing systemengaging in one or more interactive sessions with a participant. As anillustration, a participant may pick up a tablet computing device 150and select an icon for an application program that allows the user toparticipate in an interactive session.

At box 602, the computing system receives user input that specifies ausername and a security image. For example, after the participant hasselected the application icon, the computing system may present a listof usernames. The application program may have been set up or otherwiseconfigured on the tablet computing device 150 by an individual that isserving as an instructor to multiple participants, that instructor mayhave created a collection of user accounts for participants in his orher classroom, and the list of usernames presented by the tabletcomputing device 150 may represent user accounts of the participants inthat classroom. Responsive to the participant selecting his or herusername, the computing system may present a collection of images (e.g.,nine clip art drawings, including for example, one of a house, one of acar, and one of a cat), and the participant may provide input to selectone of the images in the collection.

The combination of the username and the selected security image mayserve as credentials for the participant user, and the computing systemmay compare the user-selected security image to determine whether itmatches a security image that the participant previously selected, inorder for the computing system to determine whether to provide theparticipant access to an interactive session queued up for theindividual represented by the selected username. The authenticationprocess may be relatively straightforward here (e.g., not requiring atext password and only requiring selection of a correct one of multipledisplayed security images) because participants may not be fullyliterate and may be trying to login to increase a level of theirliteracy.

At box 604, the computing system identifies an interactive sessionqueued up for the participant user. In the situation in which theparticipant user has not previously logged into the system, thatinteractive session may be the first interactive session shown in table502 of the FIG. 5 user interface. If the participant user has completedsome of those interactive sessions, the computing system may select thenext interactive session in table 502. In some implementations, theparticipant is able to select an interactive session, from among acollection of interactive sessions presented by the computing system, inwhich the participant would like to participate.

At box 606, the computing system performs a process for engaging in theinteractive session with the participant user. This process is describedin more detail with reference to FIG. 6B, which shows the set of seveninteractive exercises that the computing system will step through withthe participant user (illustrated with boxes 650, 652, 654, 656, 658,660, and 662).

Turning now to FIG. 6B, at box 650, the computing system engages in afirst interactive exercise with the user, which involves presentingcontent items to a participant and requesting that the participantrespond by speaking each of the content items. Operations for this firstinteractive exercise, termed “Flash Doodle” are described with respectto the flowchart of FIG. 7 and the user interfaces of FIGS. 8A-D.

Referring now to FIGS. 7 and 8A-D, the computing system begins the firstinteractive exercise by showing the splash screen user interface of FIG.8A, which includes seven graphical representations 802-814 of the seven,respective interactive exercises, and indicates which of those seveninteractive exercises are complete. In this illustration, no interactiveexercise is yet complete and therefore none of the “star” interfaceelements are shaded.

After the computing system presents the FIG. 8A user interface, thecomputing system presents the introductory user interface of FIG. 8B,which introduces the participant user to the name of the interactiveexercise (“Flash Doodle”) and informs the participant user of the typeof interaction involved with the interactive exercise (“Speak YourAnswer!”). The computer system presents a similar splash screen andintroductory user interface before all seven of the interactiveexercises, but this description only discusses these user interfaceshere with respect to the first interactive exercise.

Turning now to the FIG. 7 flowchart, at box 710, the computing systemselects a content item. The content item is selected from among thecontent items specified for the first interactive exercise by theadministrative user (or other authoring user, such as the instructor) inbox 422 of the FIG. 4 flowchart, as illustrated within cell 542 a oftable 550 of the FIG. 5B user interface. The content item is selected inorder. As such, if the participant user is beginning the exercise, thecomputing system selects the first content item from the list. If theparticipant repeating the operations of box 710 for a second time afterhaving completed the operations of the FIG. 7 flowchart once, then theselected content item is a second content item from the list of contentitems. In this illustration, the selected content item is the letter“t”.

At box 712, the computing system accesses user-defined text for thecontent item and visually outputs the content item. For example, theserver system 130 (FIG. 1 ) accesses information for the selectedcontent item from the content and metrics data store 120. The accessedinformation includes the information that the administrator defined forthe content item at boxes 212, 213, and 216 of the FIG. 2 flowchart,including the “text” for the content item, which in this illustration isthe single character “t.” As such, the computing system visually addsthe letter “t” to a template display into which the computing system isconfigured to render text representations of content items. An exampleillustration of the template display with a representation of the letter“t” 820 inserted therein is shown in FIG. 8C. The FIG. 8C user interfacealso includes a timer 822, which indicates an amount of time left tocomplete the interactive exercise. Also shown is a score indicator 824,which shows how many points (bananas) the user has scored during thecurrent interactive exercise. The FIG. 8C user interface also includes amicrophone button and timer 826, which counts down an amount of timethat the participant user is permitted to select the button 826 in orderto record a response to the system prompt.

At box 716, the computing system records an audible input. For example,the participant may have selected the microphone button 826 to activatethe computing system to record audio with a microphone of the computingsystem (e.g., a microphone of the tablet 150). The participant may thenhave spoken, and the computing system then records the spoken audiousing the microphone and converts the recording to a digitalrepresentation of the spoken audio.

At box 720, the computing system determines whether the recorded audiomatches the content item, the process for doing so being described inadditional detail with reference to a flowchart in FIG. 9 , to whichthis description will now turn before returning to the operations of theFIG. 7 flowchart.

At box 910, the computing system breaks the recorded audio into audiblesegments. The breaking of the recorded audio into segments involves anaudio segmenting process by which a spoken content item (e.g., an audiorecording of a word) is broken into its constituent sounds, whichenables the computing system to separately analyze each component sound.In the situation in which the content item is a single character (e.g.,just a “t” as illustrated with respect to FIGS. 7 and 8C), then thecomputing system either may skip the audio segmenting process, or mayperform the audio segmenting process with the audio segmenting processreturning the same audio that was provided to the process (althoughpotentially with “empty” portions of the recording before and after thesound of the “t” removed).

In the situation in which the content item has multiple letters (e.g.,“sat” as is illustrated with respect to FIGS. 14 and 15A), the audiosegmenting process analyzes the audio to identify each different soundin the recorded audio. The process may actually split the recording intomultiple audio files, or may simply maintain the same recording butgenerate pointers that identify where each component sound begins andends. As an illustration, the audio segmenting process could analyzeaudio of a participant speaking the word “sat” and identify the portionof the recording that corresponds to the “s” sound, the portion of therecording that corresponds to the “a” sound, and the portion of therecording that corresponds to the “t” sound. The computing system mayperform this audio segmenting process itself, or send the audio to aremote computing system for performance of the audio segmenting process.

At box 920, the computing system accesses the user definition ofcomponent sounds for the content item. For example, the computing systemaccesses the text string specified by the administrator at box 216 inthe FIG. 2 flowchart, through user interaction with the appropriatefield in column 356 of the FIG. 3 user interface. In the example inwhich the content item is a “t,” the user-defined information is simplythe text “t.” In the example of the content item “sat”, the user-definedinformation may be the text string “s,a_short,t”. The presence of one ormore separators in the text string (e.g., one or more commas or spaces)indicates that the content item includes multiple component sounds thatare separated by the one or more separators.

At box 930, the computing system selects the audible segment and thecorresponding user segment. For the illustration in which the word “sat”is being processed, the computing system would select both a portion ofthe recorded audio for the letter “s” and would also select the text “s”from the string “s,a_short,t”.

At box 940, the computing system generates a confidence score for thecomponent sound based on comparing the audible segment and thecorresponding text segment. For example, the computing system mayprovide the selected audio and text segments to an audio analysisengine. The audio analysis engine can use data libraries and/oralgorithms to determine whether a portion of recorded audio matches asound specified by the text input. For example, receipt of the audioanalysis engine of an input of recorded audio and also the text “s” cancause the audio analysis engine to run pre-configured routines todetermine whether the portion of recorded audio represents a userspeaking the sound “s”. The audio analysis engine can return aconfidence score (e.g., a number from 0.00 to 1.00) indicating alikelihood that the recorded audio matches the text.

At box 950, the computing system determines whether the confidence scoreexceeds a pre-designated threshold (e.g., 0.80). If the confidence scoredoes not exceed the threshold, at box 960, the computing system marksthe segment as incorrect or unsatisfactory, for example, by storinginformation in content and metrics database 120 that indicates that theparticipant incorrectly spoke a component sound (“s” in this example).

At box 970, responsive to either a “positive” result from thedetermination of box 950 or upon completion of box 960, the computingsystem determines whether the audio includes another segment. In theillustration in which the system is processing the word “sat,” there isanother segment after “s,” so the computing system would return to box930. Once at box 930, the computing system may select the next audiblesegment (e.g., audio for “a” in “sat”) and the corresponding portion ofthe text string (e.g., “a_short” in the text string “s,a_short,t”). Thecomputing system then repeats the operations of boxes 940, 950, and ifappropriate 960, to generate a confidence score for the next sound anddetermine whether the participant correctly spoke the next sound. Thisprocess cycles until all component sounds of the content item have beenanalyzed, and the computing system then returns to the calling process.

Returning the FIG. 7 flowchart, now at box 720, the computing systemdetermines whether the recorded audio matches the content item output bythe computing system. This determination may be performed by analyzingwhether each component sound analyzed with the process of the FIG. 9flowchart exceeded the pre-designated threshold. In this illustrationthe content item included a single component sound (i.e., the sound“t”), and therefore the analysis would be whether the audio analysisengine returned a satisfactory confidence score for a comparison of theaudio recording to the letter “t.”

At box 730, responsive to a “positive” result to the determination ofbox 720 (indicating that the participant correctly spoke the contentitem), the computing system modifies the user interface that is shown inFIG. 8C to (1) change the shading of the text representation of thecontent item to indicate a match (e.g., change the letter “t” from whiteto blue), (2) add a filled-in, colored star above the content item, and(3) animate character 828 in a reinforcing manner (e.g., with a dance).

At box 740, should the computing system determine that the audio did notmatch the content item (the “negative” branch in FIG. 7 ), the computingsystem modifies the user interface that is shown in FIG. 8C to changethe shading of the text content item indicate that there is no match(e.g., changing the letter from white to yellow), or does not change theshading of the text content item, and potentially adds some otherindication that there is no match (e.g., a star that is not filled in orcolored).

At box 742, responsive to a “negative” result to the determination ofbox 720 (indicating that the participant did not correctly speak thecontent item), the computing system stores an indication that theparticipant incorrectly input the content item. For example, thecomputing system may store, in the content and metrics data store 120,an indication that the participant incorrectly responded when promptedto speak the content item “t”.

At box 750, the computing system determines whether it is done with thefirst interactive exercise, or whether to perform the operations ofboxes 710 through 742 for another content item. The computing system maybe done with the first interactive exercise in either of at least twosituations. First, the timer 822 in the FIG. 8C user interface may haveexpired, with an initial value of the timer 822 being specified by anadministrator through user interaction with the appropriate cell incolumn 532 in the FIG. 5B user interface. Second, the system may havecycled through all available content items specified by theadministrative user for the first interactive exercise. If the timer hasnot expired and additional content items remain, the process repeatsbeginning with the operation at box 710. If the system is done, itproceeds to perform the completion routine at box 752, which isdescribed in additional detail with respect to the flowchart of FIG. 10and the FIGS. 11A-B user interfaces. After performance of the completionroutine, the computing system returns to the main interactive sessionprocess flow at FIG. 6B. This discussion now turns to FIG. 10 , whichshows a flowchart of the completion routine.

At box 1010, the computing system visually identifies multipleactivities. For example, the computing system can present the userinterface that is illustrated in FIG. 11A, which includes six interfaceelements 1110-1160 representing six different activities. At the bottomright of the FIG. 11A user interface is an indication of the currentscore 1170, which is twelve bananas in this illustration. Each of thedifferent activities requires a different score to select the associatedactivity, with the required score represented by the number at thebottom-right of each of the six interface elements 1110-1160. Allactivities are available in this illustration because the participantuser has earned twelve bananas, which is enough to select any of theactivities. Should the participant user have only earned nine bananas,for example, the system would have shown interface elements 1150 and1160 as unavailable (e.g., by shading them gray).

At box 1020, the computing system receives user input that selects oneof the multiple activities. For example, the tablet computing device 150may receive an indication that a finger or stylus contacted atouchscreen of the tablet computing device 150 at a location of thetop-left interface element 1110.

At box 1030, responsive to the user having selected the top-leftinterface element 1110, the computing system transitions to the userinterface presented in FIG. 11B, which shows an animated character 1180banging tambourines on his knees, with a central portion of the displayshowing two still images that illustrate how the participant is to joinin the activity with the animated character 1180.

FIG. 8D shows a unique type of content item that includes foursub-content items that all end with the same set of characters. Thecomputing system displays the four content items with only the same setof characters at the end of each content item presented, and anintroductory portion of each content item that differs replaced with anempty space represented by an underline. The computing system presents,one-at-a-time, a textual representation of the correct beginning foreach content item in presentation field 850 on the left side of thescreen, prompting the participant to speak a content item that beginswith the character in field 850 and ends with the characters repeated inthe center of the user interface. In this illustration, the participanthas already correctly spoken “ang” and “ing,” and is currently promptedby the presentation of “o” in field 850 to speak “ong.” Aftersuccessfully speaking “ong”, the box 850 will change to show an “e” toprompt the participant to speak “eng.” The processing of eachsub-content items is similar to the processing that the computing systemperforms for a single content item, as described with respect to theflowchart of FIG. 7 .

Returning now to the FIG. 6B flowchart, which shows the main processflow for the interactive session, with the first interactive exercise(box 650) complete the process moves to the second interactive exercise(box 652). This second interactive exercised is termed “Doodle Print”and involves the computing system requesting a participant to write acontent item rather than speak the content item, as was case with thefirst interactive session. The operations of the second interactivesession are described in detail with reference to the flowchart of FIG.12 , to which this discussion now turns.

At box 1210, the computing system selects a content item. For example,the computing system would select the initial content item defined by anadministrative user in cell 542 b in the FIG. 5B user interface. In thisillustration, the initial content item selected is a “b”.

At box 1212, the computing system accesses a user-defined recording forthe content item and audibly outputs that user-defined recording. Forexample, the server system 130 may access the content and metrics datastore 120 to retrieve a recording that an administrative user haduploaded for the selected content item, in this case a recordingrepresented in column 354 of the FIG. 3B user interface for the contentitem “s”. The computing system may then play that audio file over one ormore speakers of the computing system (e.g., a speaker of the tabletcomputing device 150). For example, the tablet computing device 150 mayaudibly output “Write ‘s’”, with the word “Write” being a templaterecording and the computing system appending the administrativeuser-defined recording to the end of the template recording.

At box 1214, the computing system presents an input field and receivestouchscreen input through interaction with that field. For example, thecomputing system may display the user interface shown in FIG. 13A, whichincludes such an input field in the center of the user interface. Insome examples, the computing system audibly prompts the user to writethe content item (as discussed above) while presenting the userinterface that is shown in FIG. 13A. The participant can respond byusing his or her finger or a stylus to write the content item into thefield (or at least the user's best attempt to write the content item).The computing system can record the user input as an animated gesture,by creating an animation or some other type of data that indicates thelocations of the user finger/stylus on the touchscreen at differenttimes, such that the computing system stores a timing and a manner inwhich the input was formed and not just a final representation of thetraced line drawing. The FIG. 13B user interface shows that the user hastraced the letter “K”, which is different than the system request towrite “s”.

In some examples, the computing system receives input that writes thecontent item by analyzing user movement in free space, for example, usermovement of a controller or their hand in free space, through computingsystem analysis of motion sensors inside of the controller or throughvideo or analysis to identify how a user is moving their hands in freespace. Such techniques may be used in implementations in which thecomputing system interacts with the participant using a virtual oraugmented reality experience, in which the content items are displayedby goggles that the participant wears or a large monitor placed in frontof the participant, and the participant provides written input by movingtheir hands in free space. Such technologies may be used in place of theuse of touchscreens discussed through this document.

At box 1216, the computing system uses a writing matching system togenerate a confidence score based on the touchscreen input and theadministrator-defined text for the content item. For example, thecomputing system may provide the touchscreen input and the text “s” to awriting matching engine, and the writing matching engine may return aconfidence score that indicates a similarity between the handwrittenletter and the administrator-defined text for the content item. In someimplementations, the writing matching system only analyzes a finalrepresentation of the user input and does not analyze an animatedgesture representative of the user input (which a different analysissystem may analyze, as described later). Here, the content item is an“s” but the user drew a “K” and the writing matching system wouldtherefore return a low confidence score, for example, 0.15 out of 1.00.

In some implementations, the computing system identifies not onlywhether the content item was input correctly, but what the participantinput if the content item was incorrectly input. For example, inaddition to returning the confidence score of 0.15 for the character“s”, the writing matching system may return a confidence score of 0.95for the character “K”. The writing matching system can also determinewhether characters in the written input are slanted forward or backward,whether characters are larger or smaller than a size designated by theinput field (e.g., whether lower-case characters improperly extend abovethe dashed line in the input field or whether uppercase characters donot extend above the dashed line in the input field). The writingmatching system can also determine whether the spacing between lettersis too large, too small, or inconsistent between letters.

At box 1220, the computing system determines whether the handwrittencontent item matched the text for the content item. The matching processmay be based on whether the confidence score exceeds or otherwisesatisfies a pre-designated threshold, such as 0.8. In this situation,the confidence score of 0.15 does not exceed the threshold of 0.8. Assuch, the computing system performs the operations of box 1232, whereashad there been a match the computing system would have performed theoperations of box 1230.

At box 1232, responsive to a “negative” result to the determination ofbox 1220, the computing system visually indicates that the touchscreeninput does not match the content item. For example, the user interfaceshown in FIG. 13C shows a text representation of the correct contentitem (“s”, not “k”), and includes a gray-shaded star above the correctcontent item rather than a yellow-shaded star, which the computingsystem would have presented had the participant written the correctcontent item.

At box 1234, the computing system stores an indication that theparticipant input the incorrect content item, for example, in thecontents and metrics data store 120. The stored indication may specifythe participant user, the interactive exercise in which the user inputthe incorrect content item, the content item that the user inputincorrectly, and a character or word that the user input instead. Thecomputing system may also store indications of any determinations by thewriting matching system that the written input was unsatisfactory, suchas the characters being too big or small, the spacing between lettersbeing too big or too small and the other similar types of determinationsdiscussed above with respect to the writing matching system.

At box 1240, the computing system determines whether to end the secondinteractive exercise or whether the second interactive exercise shouldrepeat for another content item. As described above with respect to thefirst interactive exercise, the process may continue to repeat untileither time runs out or the system cycles through all content itemsspecified by an administrative user for the second interactive exercise.

At box 1250, the computing system performs the completion routine(described in detail with reference to FIG. 10 ), before returning toFIG. 6B upon completion of the second interactive exercise.

Returning now to the flowchart in FIG. 6B, with the second interactiveexercise (box 652) complete, the process moves to the third interactiveexercise (box 654), termed “Doodle See,” which involves the computingsystem prompting a participant to speak a longer content item. Theoperations of the third interactive exercise are described in detailwith reference to the flowchart of FIG. 14 and the user interfaces shownin FIGS. 15A-B, to which the discussion now turns.

At boxes 1410, 1412, 1414, and 1420, the computing system selects acontent item, accesses text for the content item and visually outputsthat text (audibly outputting the content item in some implementation),records the participant's spoken response, and determines whether therecorded audio matches the content item. These operations are similar inmany respects to the operations of boxes 710, 712, 726, and 720 of theFIG. 7 flowchart, except that the content items are longer (e.g.,multi-letters words rather than single-letter sounds), and that thecharacteristics specified for the third interactive exercise by theadministrator are different based on the specifications of row 520 c inthe FIG. 5B user interface (e.g., the time, score, content items, etc.for the third interactive exercise).

Because the content items are longer, the audio-matching operationsdescribed with respect to the FIG. 9 flowchart may be different thanwhen the audio-matching process is performed with a single-charactercontent item. Indeed, as described above with reference to the FIG. 9flowchart, when audio recorded for a multi-character content item isanalyzed, that audio is broken into component audible segments and eachcomponent of the audio is compared to a phonetic description of thecontent item.

When the participant correctly speaks the content item, prompting theoperations of box 1430, the user interface and operations are similar tothe operations of box 730 in the FIG. 7 flowchart. For example, the FIG.15A user interface shows how the computing system indicates that a userhas correctly spoken the content item “sat”.

However, when the participant user incorrectly speaks the content item,the operations along the “negative” branch of the FIG. 14 flowchart(boxes 1440, 1442, and 1444) are different in some respects than theoperations performed for the “negative” branch during the firstinteractive exercise.

At box 1440, the computing system visually indicates that the recordedaudio does not match the content item. As an illustration, suppose thatthe content item is the term “cat” but that the participant spoke theword “bat.” In response to this incorrect entry, the computing systemmay show the user interface of FIG. 15B, which shows the star 1520unshaded to indicate that the user entry was incorrect. (Also, note thatthe animated character 1530 is not smiling.) In some embodiments, theuser interface indicates which portion of the content item the usermisspoke, with the computing system having access to such informationthrough the process described by the FIG. 9 flowchart. In theseembodiments, the computing system may present the misspoken portion in adifferent manner than a reminder of the content item (e.g., one of thethree letters may be shaded differently to indicate that the usermisspoke that one letter).

At box 1442, the computing system stores an indication that the contentitem was incorrectly input, similar to the operations that the computingsystem performed at box 740 in the FIG. 7 flowchart. A difference isthat the computing system may not only store an indication that thecontent item was incorrectly spoken, but can also store an indication ofthe portion of the content item that was incorrectly spoken (e.g., anindication that the first letter in the three-letter content item wasincorrectly spoken).

At box 1444, also different from the first interactive exercise, thecomputing system performs a first correction routine. The firstcorrection routine uses one of the re-direct content items 560 that theadministrative user specified in the FIG. 5B user interface to exposethe participant to content items that are similar to or the same as thecontent item that the user misspoke. The first correction routine isdescribed in additional detail with reference to the flowchart shown inFIGS. 16A-B and the user interfaces of FIGS. 17A-D, which are introducedbelow.

At box 1450, the computing system has completed operations for the taskthat involving presenting the first content item and analyzing aparticipant response to that presentation, and the computing system nowdetermines whether the operations of the third interactive exerciseshould be repeated for another content item (e.g., based on whether timehas run out or whether there are other content items through which tocycle). If the third interactive exercise is to finish, the computingsystem performs the completion routine of box 1460 (described inadditional detail with reference to FIG. 10 ), and then returns to theprocess flow of interactive exercises illustrated by the flowchart ofFIG. 6B.

FIGS. 16A-B show a flowchart of a process for performing the firstcorrection routine, while FIGS. 17A-D show user interfaces associatedwith the first correction routine.

At box 1610, the computing system accesses user-defined re-directcontent items. For example, the computing system can access there-direct content items 560 that an administrative user specified forthe correction routines.

At box 1612, the computing system determines one of the re-directcontent items that corresponds to the non-matching content item. In thisillustration, the participant incorrectly spoke the word “cat” and thecomputing system identifies one of the re-direct content items thatmatches the word “cat.” In some examples, the computing systemidentifies which portion of the content item the participant inputincorrectly and/or which portion of the content item the participantinput correctly, and uses that information to select the re-directcontent item. In this illustration, the user spoke “sat” instead of“cat”, and the computing system therefore finds another word that endswith the two characters “at” that the participant spoke correctly toengage the participant in additional exercises using content items thatare similar to but different from the content with which the participanthad difficulty. This illustration assumes that the re-direct contentitems include a word “mat” and therefore the computing system willselect the word “mat.”

If the user had stated “cap” instead of “cat,” the computing system mayhave selected a re-direct content item that included the letters “ca” atthe beginning of the content item, and therefore may have selected thecontent item “cab”. In some implementations, the computing system simplyselects the same word that the user incorrectly input (e.g., the contentitem “cat” here), which is what is illustrated in this example and inthe user interfaces of FIGS. 17A-B.

At box 1612, the computing system accesses user-defined text for there-direct content item, displays text for that content item in a firststyle, and places a graphical marker at a beginning writing location ofa first letter of that text. For example, the computing system maydisplay the user interface of FIG. 17A. That user interface may includea display of the text “cat” due to the administrator having previouslyentered this text (with the text presented in a first style, such asgray but with the first letter in slightly-darker gray indicating thatthe first letter is activated for user tracing), and a display of agraphical marker such as star 1710 at a beginning writing location ofthe “c”. The computing system may store, for each letter in thealphabet, information that indicates a satisfactory tracing of thatrespective letter (e.g., where the graphical star 1710 should be placedon the letter to start a tracing).

At box 1614, the computing system receives user input that contacts theletter at a location of the marker. For example, the computing systemmay receive an indication that a user has touched a touchscreen at alocation at which the touchscreen is presenting the graphical star 1710.

At box 1620, the computing system determines that the user input movesacross the display. For example, the computing system may determine thatthe user contact with the touchscreen remains in contact with thetouchscreen after having touched the graphical start 1710, and has movedacross the touchscreen without the user contact lifting from thetouchscreen.

At box 1622, the computing system determines that the user movementtracks the displayed letter, and in response moves the marker to trackthe user movement and shades the contacted portion of the letter. Forexample, the computing system may determine that the user contact tracedpart way around the “c” and therefore the computing system may move thegraphical star 1710 to correspond to the movement of the user contactand follow the tracing a portion of the way around the “c.” Thecomputing system may change a shading of the traced portion of thecharacter, as FIG. 17B illustrates with a traced portion of thecharacter “a” being shaded differently than an untraced portion of thecharacter “a”.

At box 1624, the computing system determines that the movement does nottrack the displayed letter, and in response the computing system doesnot move the marker until user input returns to the location of themarker. For example, if the user had touched the graphical star at thetop of the “c” and dragged upward rather than around the letter, thecomputing system would not have moved the marker from the location atthe top of the “c.” The user input may have had to return to contactingthe graphical star and begun tracing the letter “c” before the computingsystem would move the marker and shade the traced portion of the letter.

At box 1630, the computing system determines whether the letter iscomplete. For example, the computing may repeatedly determine whetherthe user contact has dragged the marker 1710 to an ending trace locationof the letter (e.g., ending location 1720), in which case the computingsystem would have changed the shading of the entire letter. If the userhas not completed tracing the letter, the computing system continues totrack whether the user input has moved across the display (box 1620) andeither moves the graphical star (box 1622) or does not move thegraphical star (box 1624) based on the characteristics of the usermovement.

At box 1632, the computing system has determined that the letter iscomplete, and indicates that the letter is complete. For example, thecomputing system may change the shading of an interface element 1730(e.g., a star) above the letter to indicate that the letter has beencorrectly traced.

At box 1634, the computing system determines whether the content itemincludes another letter. In this illustration, the content item “cat”includes another letter after the letter “c” and the computing systemmove the graphical marker to a starting tracing position of the nextletter, and repeats the process of boxes 1620 through 1634 for that nextletter. In some examples, the letters may have to be traced in order. Ifall letters have been fully traced, the computing system moves to theoperations of box 1640 (FIG. 16B).

At box 1640, the computing system displays another marker at a locationof a letter in the content item. This is the first iteration of box1640, so the marker would be displayed at the location of the firstletter. For example, the computing system may present the user interfaceshown in FIG. 17C, and add the interface element 1740 to the displaycentered below the location of the first letter in the content item“cat”. The interface element 1740 may be located in other locations inwhich the interface element 1740 is closer to the first letter in “cat”than any other letter in “cat” (e.g., above the first letter or touchingthe first letter). The interface element 1740, in this illustration, isa glue bottle, and the computing system outputs the audio “Glue ittogether!” to prompt the user to contact the glue bottle and drag itrightward across the touchscreen. In this illustration, the computingsystem also adds the line 1650 to the display to indicate a direction inwhich to move the glue bottle.

At box 1650, the computing system receives user input that contacts themarker and drags the marker beyond the letter. For example, thecomputing system may determine that user input contacted the touchscreenof the interface element 1740 and dragged the interface element 1740rightward past a location of the letter “c” to a location of the letter“a”.

At box 1652, the computing system displays a path traced by the marker.For example, the computing system shades a portion of the line 1750 tothe left of the interface element 1740 to indicate the portion of theline 1750 that has been traced.

At box 1654, the computing system indicates that the letter is complete.For example the computing system may change a shading of the letters “c”and “a” as the user drags the marker 1740 past those letters.

At box 1656, the computing system audibly outputs the letter. Forexample, as the user drags past each letter in the content item “cat,”the computing system may output an audio representation of that portionof the content item. For example, the computing system may play an audiorecording of the content item “cat” in pieces, playing “c” as the userdrags past the “c”, and then playing “a” as the user drags past the “a.”

To be able to match portions of an audio recording of the content item“cat” to the position of the user dragging, the computing system mayidentify the different portions of the recording of the content itemthat correspond to the different letters, using the process describedwith respect to box 910 of the FIG. 9 flowchart. The computing systemcan match the each segment of the segmented audio to the different textletters being presented on the display, and therefore play/pause theaudio recording of the content item to correspond to the speed of theuser input that “glues it together.” In some implementations, thecomputing system alternatively or additionally records audio of the userspeaking the content item as the user “glues it together,” and providesan indication regarding whether the user correctly spoke the contentitem as a whole and/or each portion of the content item.

With the third interactive exercise complete, the computing systemreturns to the main process flow of interactive exercises illustrated bythe flowchart of FIG. 6B, and at box 656 moves onto the fourthinteractive exercise, which presents text for a multi-character contentitem, prompts the participant to speak the content item, and thenprompts the participant to write the content item. The fourthinteractive exercise is described in additional detail with respect tothe flowchart of FIGS. 18A-B and the user interfaces of FIGS. 19A-B, towhich this description now turns.

The operations of boxes 1810 through 1844 represent a first portion ofthe fourth interactive exercise. The operations in this first portion ofthe fourth interactive exercise are similar to those performed withrespect to the third interactive exercise and boxes 1410 through 1444 ofthe FIG. 14 flowchart, with the computing system displaying text for acontent item, prompting a user to speak the content item, and thenproviding feedback regarding whether the user correctly spoke thecontent item. The computing system may, during the audio playing of thecontent item and the recording of the user response, display the userinterface that is presented in FIG. 19A.

As discussed below, the second portion of the fourth interactiveexercise involves writing the content item. Before discussing thewriting portion of the fourth interactive exercise, however, thisdisclosure will discuss some alternatives to this first portion of theinteractive exercise. For example, the computing system may not performthe first correction routine described with respect to box 1844 if aparticipant misspeaks a content item, as is the case with the thirdinteractive exercise. Also, an alternative implementation involves thecomputing system selecting the content item and playing an audiorecording of the content item (boxes 1810 and 1812), but then skippingto box 1850 and the writing portion of the interactive exercise, withoutrecording audible input and analyzing the audible input (boxes 1814through 1844). In other words, the computing system may only perform thesecond portion of the fourth interactive exercise.

At box 1850, now discussing the second portion of the interactiveexercise, the computing system presents an input field and receivestouchscreen input. For example, the computing system may present theuser interface that is shown in FIG. 19B and receive user input thatwrites the word “sat” on the screen.

At box 1860, the computing system uses a writing matching system togenerate a confidence score based on comparing the touchscreen input touser-defined text for the content item. For example, the computingsystem may perform the operations previously described with respect tobox 1216 of the FIG. 12 flowchart to determine whether the touchscreeninput matches the content item, for example, returning a confidencescore indicating a likelihood of a match between the written content andthe content item that the participant user was prompted to write. Thewriting matching system can return a confidence score for each letterthat the user wrote, indicating which one or more letters the user wroteincorrectly and which one or more letters the user wrote correctly. Thewriting matching system can also indicate if there are otherunsatisfactory characteristics of the user input, as described withrespect to the FIG. 12 flowchart

At box 1862, the computing system determines whether the written inputmatches the content item, for example, by determining whether theconfidence score exceeds a threshold value.

At box 1870, responsive to determining that the touchscreen inputmatches the content item, the computing system visually indicates thatthe touchscreen input matches the content item. The computing system maydo this by replacing the written content illustrated in FIG. 19B with atext representation of the content, similar to that shown in FIG. 13C,but with the star at the top of the display shaded to indicate a correctmatch (and the animated character 1350 smiling).

At box 1880, responsive to determining that the touchscreen input doesnot match the content item, the computing system visually indicates thatthe touchscreen input does not match the content item. For example, thecomputing system may replace the written content illustrated in FIG. 19Bwith a text representation of the content, similar to that shown in FIG.13C, but in this “negative result” scenario, to indicate that the resultwas not correct, the star at the top of the display may not be shaded,the animated character 1350 may not smile, and the computing system mayhighlight one or more letters that the computing system determined theparticipant to incorrectly input.

At box 1882, the computing system can store an indication that theparticipant incorrectly input the content item. For example, the serversystem 120 may store information identifying the content item, thefourth exercise, and which one or more letters the user wroteincorrectly in the content and metrics data store 120.

At box 1886, the computing system performs a second correction routine,which engages the participant in additional actions to expose theparticipant to the content item that the participant incorrectly wrote,or similar content items. The operations of the second correctionroutine are discussed below with respect to FIG. 20A, but thisdisclosure first describes the last few boxes of the flowchart of FIGS.18A-B.

At box 1872, regardless whether the user correctly or incorrectly wrotethe content item, the computing system performs a gesture-comparisonprocess on the touchscreen input. This process analyzes how the userwrote the content item, rather than just whether the written inputultimately matched the content item, which was a focus of box 1860. Theoperations of this gesture-comparison process are described below withrespect to FIG. 22 , but this disclosure first describes the last fewboxes of the flowchart of FIG. 18A-B.

At box 1890, the computing system determines whether the fourthinteractive exercise is complete. As discussed with the otherinteractive exercises, the interactive exercise may be complete when theallocated time has completed or the system has cycled through all thecontent items specified by the administrator for the interactiveexercise. If the interactive exercise is not complete, the systemreturns to box 1810 to repeat the operations of the FIG. 18A-B flowchartfor a next content item. If the interactive exercise is complete, thesystem performs the operations of box 1892 to perform the completionroutine (described with reference to FIG. 10 ), and then returns to themain interactive exercise process flow of FIG. 6B.

This description turns now to the flowchart of FIGS. 20A-B and the userinterfaces of 21A-F, which describe the second correction routine, whichthe computing system requests at box 1844 when a participant incorrectlywrites a content item, and which exposes a participant to the same orsimilar content items.

At box 2010, the computing system accesses the user-defined re-directcontent items. For example, the computing system can access there-direct content items 560 shown in the FIG. 5B user interface, asspecified by an administrator during a process for defining theinteractive session. This collection of re-direct content items may bethe same as that from which the first correction routine of FIGS. 16A-Bpulls.

At box 2012, the computing system determines a re-direct content itemthat corresponds to the non-matching content item. Similar to theoperations of box 1612 in the FIG. 16A flowchart, the computing systemmay have identified which one or more letters the participantincorrectly wrote and which one or more letters the participantcorrectly wrote. The computing system may search through the collectionof re-direct content items 560 to identify a content item that includesthe same letters in the same position that the user correctly wrote, forexample, identifying a word that is similar to the word that theparticipant spelled incorrectly but that swaps out the one or moreletters that the user misspelled with different one or more letters. Ifthe system does not find such a word within the collection of contentitems 560, the computing system may pick a word that includes as manysimilar characters as possible, or selects the content item that theparticipant incorrectly wrote. Alternatively, the system does not selectone of the re-direct content items 560 and simply uses the content itemthat the participant incorrectly wrote.

At box 2014, the computing system accesses user-defined text for there-direct content item and displays text for the re-direct content itemin a first style. For example, the server system 130 may access thecontent and metrics data store 120 to select the text that theadministrator defined for the content item when creating or editing thecontent item via the user interfaces of FIGS. 3A-B. The computing systemmay then display that text in a first style. The computing system atthis point may prompt the user (e.g., by audio or with text) to “Tap toSpell” the letters.

At box 2020, the computing system receives user input that contacts anactive letter. For example, the computing system may activate the firstletter in FIG. 21A (i.e., the letter “c”) by shading that letter in acolor or pattern that is different than other of the letters. Thecomputing system may then determine that the user has tapped theactivated letter, for example, by determining that user input touchedthe touchscreen at a location at which the activated letter isdisplayed.

At box 2022, the computing system may indicate that the user hasselected the letter. For example, the computing system may (1) changethe shading for that letter, (2) highlight a “star” interface elementabove the letter, and (3) activate a next letter in the content item bychanging the shading of that next letter. Should the computing systemreceive user input that selects a non-activated letter or a region ofthe display not associated with the activated letter, the computingsystem may not perform the above-described operations to indicate that auser has selected a letter.

At box 2024, the computing system may audibly output the selectedletter. For example, after the computing system determines that userinput has selected the letter “c”, the computing system may not onlyshade that letter, highlight the star above that letter, and activatethe next letter, but the computing system may also play a recording ofthe letter, for example, by playing a recording of the letter “c” withspeakers. This recording may be one specific to the letter, and may notbe a segmented portion of a recording of the entire word, as may be thecase with the “Glue it Together” operations described with respect toboxes 1640 through 1662 of the FIG. 16B flowchart, which describes thefirst correction routine. Alternatively, the computing system may parsea recording of the entire word into its component portions and outputeach portion, as described with respect to box 1656 of the FIG. 16Bflowchart.

At box 2030, the computing system determines whether the content itemincludes another letter. For example, if the participant has tapped allletters and the computing system has audibly output all letters, thecomputing system takes the “negative” path branching off box 2030. Ifthe participant has not selected all letters yet, the computing systemtakes the “positive” path branching off box 2030 to perform theoperations of boxes 2020 through 2030 until the participant has selectedevery component/letter of the content item.

At box 2032, the computing system prompts the user to speak the contentitem letter-by-letter, records audio input, and indicates whether theuser spoke the content item correctly. For example, the computing systemmay present the user interface shown in FIG. 21C, and as the user speakseach letter in order, the computing system may change the shading ofeach respective letter until the user has spoken each letter. This issimilar to the above-described “tap each letter to spell,” except thatit involves speaking each letter. In some implementations, the computingsystem does not prompt the user to speak each letter one at a time,thereby skipping box 2032.

At box 2040, the computing system presents an input field and receivestouchscreen input. For example, the computing system may present theuser interface that is shown in FIG. 21D. The participant, havingfamiliarized himself or herself with the spelling of the content item,at this point is to write the content item within the input field 2150.FIG. 21E shows the user interface after the computing system hasreceived the handwritten user input and displayed that writing in theinput field.

At box 2042, the computing system uses the writing matching system togenerate a confidence score based on the touchscreen input and theuser-defined text. For example, as described with respect to box 1216 ofthe FIG. 12 flowchart and box 1860 in the FIG. 18B flowchart, thecomputing system may send, to a writing matching engine, both thecontent that was written into the text field (potentially without ananimation showing how the input was written in real-time) and the textfor the content item, and the writing matching engine may return (1) anoverall confidence score for the entire content item, (2) a confidencescore for each individual letter of the content item, or (3) both anoverall confidence score and a confidence score for each letter.

At box 2050, the computing system determines whether the written inputmatches the content item, for example, by determining whether theoverall confidence score exceeds a threshold value, or whether theconfidence score for each letter exceeds a threshold value.

At box 2060, responsive to a “positive” result to the determination ofbox 2050, the computing system presents the user-defined text with anindication that the content item was spelled correctly. For example, thecomputing system may replace the user interface that is shown in FIG.21E (showing the participant-written content item “can”) with the userinterface that is shown in FIG. 21F (showing machine-generated text forthe content item “can”, and filling in a star at the top of the inputfield).

At box 2070, responsive to a “negative” result to the determination ofbox 2050, the computing system presents the user-defined text with anindication that the content item was not spelled correctly. For example,the computing system may show the user interface of FIG. 21F, butwithout filling in the star at the top of the input field, whichindicates that the content item was not input by the participantcorrectly.

The above-discussed first correction process (see FIGS. 16A-B) andsecond correction processes (see FIGS. 20A-B) serve to re-direct aparticipant if the computing system has determined that the participantprovided an unsatisfactory input. The re-direction may occur after aparticipant has provided an unsatisfactory input one or more times(e.g., three times in some examples), and the re-direction provides theparticipant additional exposure to content with which the participant ishaving difficulty, rather than simply moving the student to the nextcontent item in the queue.

In some implementations, the computing system can escalate theparticipant to different re-direction routines if the participantcontinues to provide unsatisfactory input. One example other type ofre-direction routine can involve a mindfulness exercise in which thecomputing system provides the participant with relaxing sounds and/or arelaxing visual environment. Another example re-direction routine caninvolve a mobility exercise, for example, that described with respect tothe completion routine (see FIG. 10 ). Another example re-directionroutine can involve the computing system engaging the participant inreading, writing, and/or speaking exercises involving single-lettercontent items. As mentioned above the computing system can change thetype of re-direction routine provided to a participant if theparticipant continues to provide unsatisfactory inputs.

This discussion now turns to a gesture-comparison process, theoperations of which are shown in the flowchart of FIG. 22 . This processwas introduced above during the discussion of the fourth interactiveexercise, and is requested at box 1872 in the FIG. 18B flowchart. Insome implementations, the gesture-comparison process may be performedafter every handwritten participant input, for example, also during thesecond interactive exercise (see FIG. 12 ) and during the fifthinteractive exercise, as described below.

At box 2210, the computing system separates the touchscreen input intoanimated gestures for the individual letters. The computing system mayinitially access an animated gesture that represents the movement of aparticipant-written input entering multiple characters. As discussedabove, the term animated gesture is used for clarity and does notrequire that the computing system store a video animation of the userinput, but rather stores any form of data that indicates the location ofthe user input at different points in time, in contrast to a staticimage that simply shows the final result of the user-traced input. Asingle animated gesture can represent user input lifting from thetouchscreen and coming back into contact with the touchscreen and doesnot require that the user input be a continuous line. The computingsystem may then run or request that another process identify theportions of the participant-generated animated gesture that correspondto each letter. In some implementations, the separation process createsa separate animated gesture representing each character within theoriginal gesture. In some implementation, the separation process createspointers that identify the beginning and ending times or locations ofeach letter in the original animated gesture.

At box 2212, the computing system selects an initial letter. Theselected letter may be a text representation of the first letter writtenby the participant, even if the first-written letter was incorrect. Inother words, the accessed information may be an output of the writingmatching system 1860, which can generate a textual representation of theuser input even if the user input incorrectly writes the first letter.

At box 2214, the computing system accesses a set of pre-stored animatedgestures for the selected letter from among a plurality of animatedgestures stored for multiple different letters. The set of animatedgestures are represented in FIG. 22 by the pre-stored animated gesturesgraphic 2216, although the set of pre-stored animations may be stored inthe content and metrics data store 120 that is illustrated in FIG. 1 .The plurality of pre-stored animated gestures may include a set ofmultiple animations for each letter of the alphabet (e.g., a set of atleast two, three, four, or more animated gestures for each of thetwenty-six letters in the English alphabet). Data stored with orotherwise associated with the animated gestures may indicate that oneanimated gesture in each set of animated gestures is designated as arepresenting a satisfactory technique for drawing the letter representedby the animated gesture (e.g., an animated gesture that starts drawingan “s” at the top of the letter and correctly traces to the end of theletter). The other aminations may represent unsatisfactory processes todraw the letter (e.g., start the “s” at the bottom of the letter, orwrite the “s” backwards).

At box 2220, the computing system compares the touchscreen input of theselected letter to a pre-stored animated gesture for the selectedletter. As an illustration, the computing system may have identifiedboth (1) an animated gesture or a portion of an animated gesture thatrepresents the participant writing a single letter (e.g., excludingwriting another letter), and (2) a first, pre-stored animated gestureselected from among the accessed set of pre-stored animated gestures. Inthis example, the first animated gesture represents a satisfactorytechnique for writing the letter. The computing system may request thata movement-comparison process compare the animated gesture thatrepresents the user writing the first letter to the selected, first,pre-stored animated gesture that represents writing the first letter.

The movement-comparison process may correlate the times that it takeseach of these gestures to form the letter, so that even if the gesturesform the letter at different speeds, the techniques can be compared toeach other. For example, the computing system may normalize a length oftime represented by each animated gesture so that both begin atapproximately the same time and end at approximately the same time. Thiscorrelation/normalization process can involve scaling the time values ofone or both of the animated gestures, or not scaling values but leavingthe values in their original form but mapping times from theparticipant-formed gesture to the pre-stored gesture.

The movement-comparison process may then, after the times of theanimated gestures have been correlated, and for each of multiple momentsin time (e.g., at 20 evenly-spaced apart times during the formation ofthe gestures), determine how far a location of the user input for theparticipant-defined gesture is from a location representing user inputfor the pre-stored gesture. For example, the computing system maydetermine, 10% of the way into the formation of each animated gesture, adistance between a single point of current user input in theparticipant-generated animated gesture and a single point representinguser input in the pre-stored animated gesture. The computing system maydetermine a similarity score for each moment in time based on thisdistance.

The computing system may aggregate all of the similarity scoresgenerated for the various moments in time to generate a confidence scorethat indicates a similarity in how the two animated gestures wereformed. As such, if a participant started drawing a letter from a bottomof the input field and a pre-stored animation represents user inputstarting the drawing of the letter from the top of the input field, theconfidence score is likely to be low because the locations representingthe user inputs will only be near the same portion of a commonly-sizedinput field in the center of the letter where the animated gesturesapproximately cross each other. Similarly, if the participant-definedanimation was written backwards, the input locations would differ fromthose of a pre-stored animation that was “written” in a more up-and-downmanner (although with a higher confidence score than the example inwhich the participant-supplied gesture and pre-stored gesture started ondifferent ends of the letter).

In some implementations, the computing system normalizes not only thelength of time it takes to perform the participant-created andpre-stored animated gestures, but also a size/scale of the animatedgestures. Still, it may be beneficial to determine that a participant isinputting letters larger than expected or smaller than expected, oroffset from an expected location. As such, in some implementations, thecomputing system may not normalize the size/scale of theparticipant-created and pre-stored animated gestures to be similar insize/scale. In some implementations, the computing system may normalizean offset of the participant-specified animation so that the resultingletter is centered within a region of comparison (assuming that thepre-stored animated gesture is similarity centered within its own,respective region of comparison). The regions of comparison may be thesame size.

At box 2222, the computing system determines whether theparticipant-defined animated gesture matches the pre-stored animatedgesture. For example, the computing system may determine whether athreshold score that results from the comparison process of box 2220exceeds a threshold value.

At box 2230, responsive to a “positive” result of the determination ofbox 2222, the computing system determines whether the pre-storedanimated gesture that matched the participant-defined animated gesturerepresented a satisfactory technique in which to write the letter, forexample, by accessing information stored with the pre-stored animatedgesture.

At box 2240, responsive to a “positive” result of the determination ofbox 2222 (indicating that the matching animated gesture represents asatisfactory way to write the letter), the computing system determineswhether the gesture-comparison process of FIG. 22 has another letter toanalyze in the content item. If no, the gesture-comparison process iscomplete and the computing system returns to the process that called thegesture-comparison process. Otherwise, the operations of FIG. 22 repeatfor another letter.

At box 2224, responsive to a “negative” result of the determination ofBox 2222 (indicating that the participant-defined animated gesture didnot match the pre-stored animated gesture), the computing systemdetermines whether the set of pre-stored animated gestures accessed forthe selected letter includes one or more additional, animated gesturesthat have not yet been compared to the participant-defined animatedgesture. If not, the computing system moves to box 2240, to determinewhether there are additional letters in the content item to analyze.

At box 2226, responsive to a “positive” result of the determination ofbox 2224 (indicating that there are one or more pre-stored animatedgestures still to compare to the participant-defined animated gesture),the computing system selects a next animated gesture from the set ofpre-stored animated gestures and returns to box 2220 to perform thecomparison process described above for the next animated gesture.

At box 2232, responsive to a “negative” result of the determination ofbox 2230 (indicating that the process has identified a matching animatedgesture, but that the matching animated gesture represents anunsatisfactory mechanism for drawing the letter), the computing systemstores an indication of the unsatisfactory animated gesture that wasdetermined to match the participant-defined animated gesture. Storingthe indication may include setting a flag or recording a setting thatspecifies that the participant-defined animated gesture matches aparticular method of writing a letter (or incrementing a counterreflecting a number of times that the participant has used a particulartechnique to write a letter, such as having written five lettersbackwards).

Example animated gestures in the set of animated gestures can include,for example, (1) an animated gesture that writes the letter on a slant,(2) an animated gesture that is oversized for the space provided, (3) ananimated gesture that starts the letter in a different portion of theletter than in the satisfactory animation (e.g., at an opposite side ofthe letter such as the bottom rather than the top), (4) an animatedgesture that writes the letter in the opposite case (e.g., uppercase vs.lowercase), and (5) an animated gesture that writes the letterbackwards. The stored indication can specify, instead of or in additionto how the pre-stored animated gesture is different than theparticipant-defined animation, information that explains a potentialreason that the pre-stored animated gesture may be different than theparticipant-defined animated gesture, such as an indication that abackwards-written letter may suggest “dyslexic tendencies.”

In some implementations, the computing system does not determine, at theend of the above-described comparison process between theparticipant-defined animated gesture and a pre-stored animated gesture,whether the confidence score indicates that the process can terminate.Rather, the computing system may compare the participant-definedanimated gesture to every one of the pre-stored animated gestures in theset of pre-stored animated gestures that the computing system accessedfor the selected letter. The computing system may then identify which ofthe pre-stored animated gestures has a highest confidence level amongthe set, or which of multiple pre-stored animated gestures haveparticularly high confidence levels (e.g., both a pre-stored animationthat represents slanted input and a pre-stored animation that representsoversized input).

Even if multiple pre-stored animated gestures exceed a threshold and/oran unsatisfactory pre-stored animated gesture has a confidence scorehigher than that for a satisfactory pre-stored animated gesture, thecomputing system may only flag the unsatisfactory, pre-stored animatedgesture as a matching gesture if the confidence score for thesatisfactory pre-stored animation fails to satisfy a threshold value. Inother words, if the participant-defined animated gesture sufficientlymatches a pre-stored animated gesture that is designated as beingsatisfactory, the computing system may not take into account whether theparticipant-defined animated gesture also matches some of theunsatisfactory pre-stored animated gestures.

The gesture-comparison process can also provide a convenient time togenerate other metrics that involve analysis of the participant-suppliedinput. For example, the computing system may have received from thewriting matching system an indication of the text that theparticipant-generated input represents. The computing system can thencompare the text representation generated by the writing matching systemto determine whether the participant-supplied input includes (a) anextra letter inserted, (b) a letter missing, and/or (c) letters orderedincorrectly. If the computing system determines that theparticipant-supplied input matches any of these conditions, thecomputing system may store data indicating such. For example, thecomputing system may store an indication of the content item that theparticipant was supposed to write, and an indication of what theparticipant actually wrote. In some implementations, these comparisonsdo not involve comparing a participant-defined gesture to pre-storedgestures, and rather may simply involve analysis of the final, writtenrepresentation of the user input and the content item that theparticipant was prompted to write.

˜Now returning to the main interactive session flow of FIG. 6B, thecomputing system has completed the fourth interactive exercise (Box656), and moves onto the fifth interactive exercise (Box 658). The fifthinteractive exercise is described with reference to the flowchart ofFIGS. 23A-B and the user interfaces of FIGS. 24A-B, and involves thecomputing system outputting a recording of a multi-word content item,receiving user input that writes into an input field, and analyzing theuser input to determine if the user input matches the multi-word contentitem.

At box 2310, the computing system accesses a collection of content itemsspecified by an administrator for the fifth interactive exercise, forexample, through previous administrator interaction with field 542 e ofthe FIG. 5B user interface. As mentioned above, each of these contentitems may represent multiple-words, and may include a capital letter atthe beginning of the first word and a punctuation mark at the end of thelast word, and therefore the content items may be referred to assentences.

At boxes 2312, 2314, 2316, and 2318, the computing system accesses arecording of the content item, outputs the recording, receivesparticipant user input that writes into an input field, and uses awriting matching system to compare the participant-defined input to textdefined by the administrator when the administrator originally createdand defined the content item. FIG. 24A shows an example user interfaceafter a user has written the words “man can” into an input field. Theseoperations are similar to those of boxes 1812, 1850, and 1860 of theFIGS. 18A-B flowchart, which represents the fourth interactive exercise,and will not be described further here to avoid repetition.

At box 2330, the computing system determines whether theparticipant-written input matches the content item, for example, bydetermining whether a confidence score generated by the writing matchingsystem of box 2318 exceeds a threshold value.

At box 2332, responsive to a “positive” result of the determination ofbox 2330 (indicating that the administrative-written input matches thecontent item), the computing system visually indicates that thetouchscreen input matches the content item.

At box 2340, responsive to a “negative” result of the determination ofbox 2330 (indicating that the participant-written input does not matchthe content item), the computing system visually indicates that thetouchscreen input does not match the content item. For example, the userinterface of FIG. 24B may not shade the star 2450 at the top of the userinterface, to indicate to the participant that the written input did notmatch the content item that the computing system audibly output.

At box 2342, the computing system may indicate that a sub-set of thecontent item is incorrect. As an illustration, the content item may havebeen the sentence (“Man can.”) and the participant-written input mayhave been determined to be the phrase (“Man can”), but without theperiod at the end. The writing matching system of box 2318 may havereturned a confidence score for each letter/portion of the content item,including for the period punctuation mark, and the confidence score forthe punctuation mark may have been below a threshold level, indicatingthat the participant-written input did not include a punctuation mark orthat the participant-supplied punctuation mark was unsatisfactory. As aresult, in the user interface that is shown in FIG. 24B, the computingsystem can display the correct phrase (“Man can.”) but may shade theword “can.” differently than the word “Man” to visually indicate to theparticipant which portion of the participant-supplied input thecomputing system determined to not match the originally-presentedcontent item.

At box 2344, the computing system can store an indication of the contentitem that the computing system determined that the participantincorrectly input, similar to the operations of box 2072 from the FIGS.20A-B flowchart.

At box 2350, regardless whether the participant-written content matchesthe content item or not, the computing system performs thegesture-comparison process that is described in additional detail withrespect to FIG. 22 . As described above, this process parses througheach letter (and punctuation mark) of the participant-written input, andcompares the manner in which the participant formed a gesture thatdefines the letter to previously-defined and stored animated gestureswhich represent various techniques for forming the letter. Thepre-stored animated gestures represent various satisfactory andunsatisfactory techniques for forming the letter, and the computingsystem steps through comparisons of the participant-defined gesture toone or more of these pre-stored animations to determine whether theparticipant-defined input is satisfactory or unsatisfactory. Ifunsatisfactory, the system can indicate how the participant-writteninput was unsatisfactory, for example, by flagging a setting indicatingthat the participant wrote a letter that was, for example, “backwards”or “starts at an improper location.” The computing system may store theindication in association with the content item that the participantinput incorrectly.

At box 2360, the computing system determines whether the interactiveprocess is to be done, for example, whether a timer has elapsed or thesystem has cycled through all content items. Responsive to a “negative”result to the determination regarding whether the interactive process isdone, the computing system selects a next content item from the orderedset specified by an administrative user, and returns to box 2310 toperform the operations of FIGS. 23A-B again for the next content item.

At box 2370, responsive to “positive” result to the determination of Box2360 (indicating that the interactive exercise is to be done), thecomputing system performs the completion routine (Box 2370), which isdiscussed in detail with reference to FIG. 10 .

Now returning to the main interactive session flow of FIG. 6B, thecomputing system has completed the fifth interactive exercise and movesonto the sixth interactive exercise (box 660). The sixth interactiveexercise is described with reference to the FIG. 25 flowchart, andinvolves the computing system providing a preview of content items forthe next interactive session, by performing various different types ofinteractive exercises with the previewed content items.

At box 2510, the computing system performs a single iteration of thefirst interactive exercise or the second interactive exercise. Forexample, the computing system may select one of the content itemsspecified by an administrator in cell 542 g from the FIG. 5B userinterface, which includes single-letter content items, and performeither the first interactive exercise or the second interactive exerciseusing the selected content item. The performance of the single iterationof the first interactive exercise may include performing the operationsof boxes 712 through 730 or 742 of the FIG. 7 flowchart using theselected content item (depending on the branch taken at decision box720), without performing the completion routine of box 752 (whichrequests performance of operations in the FIG. 10 flowchart). Theperformance of the single iteration of the second interactive exercisemay include performing the operations of boxes 1212 through 1230 or 1234from the FIG. 12 flowchart using the selected content item (depending onthe branch taken at decision box 1220) without performing the completionroutine of box 1250 (FIG. 10 ). The computing system can randomly selectwhether to perform the first interactive exercise or the secondinteractive exercise, or can perform both interactive exercises for theselected content item.

At box 2520, the computing system performs a single iteration of thethird interactive exercise. For example, the computing system may selectone of the content items specified by an administrator in cell 542 h ofthe FIG. 5B user interface, and may use the selected content item toperform the operations of boxes 1412 through either 1430 or 1442 fromthe FIG. 14A flowchart, depending on the branch taken at decision box1420.

At box 2530, the computing system performs a single iteration of thefourth interactive exercise. For example, the computing system mayselect one of the content items specified by an administrator in cell542 i of the FIG. 5B user interface, and may use the selected contentitem to perform the operations of boxes 1812 through 1872, excluding thecorrection routine boxes 1844 and 1886 in the flowchart of FIGS. 18A-B,depending on the branch taken at decision box 1862.

At box 2540, the computing system determines whether the interactiveexercise is to be done, for example, due to time running out or thecomputing system exhausting all content items. If not, the computingsystem repeats the operations of boxes 2510 through 2530, skipping a boxif there are no more content items available for the correspondinginteractive exercise.

In some implementations, the computing system performs multipleiterations of the first and/or second interactive exercises (box 2510)before performing multiple iterations of the third interactive exercise(box 2520) and then performing multiple iterations of the fourthinteractive exercise (box 2530). The computing system may repeat each ofthe above-described boxes until the time limits specified in the cellsof column 532 (FIG. 5B) expires or the content items specified in therespective cell of cells 542 g-i (FIG. 5B) are used up.

At box 5550, the computing system performs the completion routinedescribed with respect to FIG. 10 .

Now returning to the main interactive session flow of FIG. 6B, thecomputing system has completed the sixth interactive exercise, and movesonto the seventh interactive exercise (box 662). The seventh interactiveexercise is described with reference to the flowchart of FIG. 26 , andinvolves the computing system providing the participant with a matchingexercise.

At box 2610, the computing system selects four content items. Forexample, the computing system may select any four single-character orsingle-word content items from those shown in the user interface of FIG.5B, although in some implementations the user interface of FIG. 5Bincludes a field in which an administrator can specify the content itemsfor the seventh interactive exercise.

At box 2620, the computing system displays a grid of eight cards withthe faces down, the each of the selected content items being representedon two respective instances of the eight cards (of course, with thecards displayed face down, the user interface does not show the contentitems on any of the cards). As an example, the user interface in FIG.27A shows a grid of eight such cards face down.

At box 2630, the computing system receives user input that selects twocards. For example, the computing system may receive two user inputsthat touch a touchscreen at two locations at which the touchscreen isdisplaying two different cards. In response, the computing system maydisplay the two selected cards face up, as illustrated by the userinterface in FIG. 27A.

At box 2640, the computing system determines whether the two selectedcards match each other, for example, by showing and being assigned tothe same content item.

At box 2650, responsive to a “negative” result of the determination ofbox 2640 (indicating that the cards are not a match, as is the case inthe FIG. 27D user interface), the computing system displays the cardsface down again and returns to the operation of box 2630 and awaits userinput that selects two face-down cards.

At box 2660, responsive to a “positive” result of the determination ofbox 2640 (indicating that the cards match), the computing system leavesthe cards in the face-up configuration, as illustrated in the userinterface of FIG. 27C.

At box 2670, the computing system determines whether the seventhinteractive exercise is complete. The exercise may be complete wheneither a timer has expired or all cards have been turned face up.Responsive to a determination that the exercise not yet complete, thecomputing system returns to box 2630 and awaits user input that selectstwo face-down cards.

At box 2680, responsive to a determination that the interactive exerciseis complete (which is illustrated in the user interface of FIG. 27D),the computing system performs the completion routine that is explainedwith respect to FIG. 10 , and then returns to the main interactivesession process flow of FIG. 6B.

Now returning to FIG. 6B, with all of the interactive exercisescomplete, the computing system returns to FIG. 6A and moves on to box608.

At box 608, the computing system provides the user with a virtual gift.The virtual gift may be provided to a participant every time that theparticipant completes an interactive session (i.e., has stepped throughall seven exercises that together comprise the whole interactivesession). An example user interface through which the computing systemshows the virtual gift to the participant is shown in FIG. 30A. In thisillustration, the computing system has awarded the participant tank top3010. This and other virtual gifts allow the participant to customizethe characteristics of their character 3020, and provide an incentive tocomplete the exercises.

At box 610, the computing system allows the user to customize theircharacter. FIG. 30B shows an example user interface with whichparticipants are able to customize the characteristics of theircharacter. In this illustration, the user is able to select from four ofsix clothing tops (i.e., any of tops 3014, 3016, 3018, and 3010) byselecting any of those four clothing tops, with the recently-gifted tanktop 3010 included as one of the four options. Options 3012 and 3020 areunavailable because they were not one of the default clothing topsavailable to the participant and the user has not yet been gifted eitherof those clothing tops. In this illustration, the user has not selectedto dress his character with the newly-gifted clothing top 3010, but hasinstead selected the jacket 3018. User interface tabs 3030 a-f allow theuser to view options to change other character characteristics, such ascharacter color 3030 a, character hat 3030 b, clothing bottom 3030 d,character footwear 3030 e, and character accessory 3030-f.

At box 620, the computing system determines whether to cycle throughanother interactive session that repeats the seven interactive exercisesdiscussed above, but with different content items. As an example, theparticipant may be provided with an option to either leave theapplication program that provides the interactive session, or to start anew interactive session. Additionally or alternatively, the number ofinteractive sessions that the computing system prompts a participant toundertake may be defined by an instructor to which the participant isassigned. FIG. 5A shows an example ordering of the interactive sessions,although the instructor can specify a different ordering of theinteractive sessions.

Analysis of Interactive Sessions

The flowcharts of FIGS. 28A-B show various user interfaces that areaccessible to an individual with administrative and/or instructor accessto the computing system described herein.

Beginning at FIG. 28A, at box 2810, the computing system provides agraphical interface to show metrics on groups of users. For example, thecomputing system may present the user interface of FIG. 29A, which auser may access by selecting the “Report” interface element in the leftpanel 310 shown in the FIG. 3A user interface (or any other userinterface that includes the left panel 310). The user interface of FIG.29A includes various different interface controls for specifying afiltering of participant data, including a start date control 2902, anend date control 2904, a type of data control 2906, a school control2908, and a student-selection control 2910.

At box 2820, the computing system receives user input specifyingsettings. For example, the computing system may receive user input thatinteracts with the above-described controls to specify start and enddates to specify a time range (box 2822), a particular type of metric toview from among multiple different types of metrics (box 2824), a groupof users on which to view metrics, such as a particular school (box2826), and whether to view metrics on all students in that group or ananonymized subset of that group (e.g., a class within a school, giventhat the student designation is anonymized along with all otherpersonally identifiable information) (box 2828).

At box 2830, the computing system presents metrics. The metrics may beshown at a display portion 2920 in the FIG. 29A user interface. Thereare various different types of metrics that the system may display,based on the metric specified by the user through interaction withcontrol 2906. These metrics may include Spelling-Decoding/Encoding (box2832), which presents metrics on missing letters, random letterinsertion, and incorrect letter ordering. These are the metrics that areillustrated in the display portion 2920 of the FIG. 29A user interface,which shows bar graph bars that specify either the amount of or thenumber of students providing (1) occurrences of missing letters (see bar2922), (2) random letter insertions (see bar 2924), and (3) incorrectletter orderings (see bar 2926).

The computing system is configured to present other metrics in thedisplay portion 2920 of the FIG. 29A user interface. For instance, userselection to view a Reading Phonology option with control 2906 causesthe computing system to show bars with statistics on the number of usersand/or occurrences of missed words, extra words, and missed sight words.User selection of a Fine Motor Skills option using control 2906 causesthe computing system to show bars with statistics on letters beingdisconnected, letters extending out, and letters being unreadable. Othermetrics may be presented, such as those determined through thegesture-comparison process of FIG. 22 , although in some implementationsthere are certain ones of the metrics that are not available to anindividual with administrative access, and rather such metrics may beviewable only by an instructor of the participants. The user interfacesfor displaying these other metrics may otherwise be the same as orsimilar to those described with reference to the FIG. 28A-B flowchartsand the FIG. 29A and FIG. 29C user interfaces.

Now turning to the FIG. 28B flowchart, at box 2850, the computing systemprovides a graphical interface to show metrics on individual users. Anexample user interface is that shown in FIG. 29C, which includes a startdate control 2932, an end date control 2934, a group-selection control2936, a participant-selection control 2938, and a display portion 2940.

At box 2860, the computing system receives user input that specifiessettings. For example, the computing system may receive user input thatinteracts with the start date control 2932 and the end date control 2934to specify a time range (box 2862). The computing system may also oralternatively receive user input that that interacts with thegroup-selection control 2936 and the participant-selection control 2938to select a group (e.g., a school) and possibly a sub-group of one ormore participant users within that selected group (e.g., a class ofstudents or an individual student, represented by anonymized identifier)(box 2864).

At box 2870, the computing system presents metrics on the selected groupof one or more students for the selected time range. For example thecomputing system can present, for each instance of a participant userperforming an interactive session within the selected time range, theproficiency of the participant on each of the exercises. As anillustration, the first row of the table shown in display portion 2940shows a name of the group to which the participant belongs (column 1),the anonymized identifier for the participant (column 2), the date thatthe participant engaged in the interactive session (column 3), the nameof the interactive session (column 4), overall proficiency on all theexercises as a percentage (column 5), and percentage correct andabsolute number correct for each of the seven interactive exercises(e.g., with column 6 showing percentage for the first exercise, column 7showing absolute number for the first exercise, column 8 showingpercentage for the second exercise, as so forth). The server system 130may access this information from the content and metrics data store 120,with this information having been stored in the data store 120 duringeach interactive exercise or at the completion thereof.

At box 2872, the computing system presents a list of the incorrectlywritten or read single-character content items. For example, thecomputing system may present, for each row in the table in FIG. 29C(representative of a single interactive session for a single user), alist of the single-character content items that the respective usereither spoke or wrote unsatisfactorily during the respective session,which is shown in FIG. 29C under column 20.

At box 2874, the computing system presents a list of the incorrectlywritten or read multi-character content items. For example, thecomputing system may present, for each row in the table in FIG. 29C, alist of the multi-character content items that the respective usereither spoke or wrote unsatisfactorily during the respective session,which is shown in FIG. 29C under column 21.

The server system 130 may access the incorrectly spoken or writtencontent items from the content and metrics data store 120. Presentingthis information can help an instructor determine if there is a specifictype of content item on which a single participant repeatedly performsunsatisfactorily, and can allow the instructor to interact with thatparticipant in a real-world training session that focuses on areas ofthat participant's literacy that may need additional attention.Moreover, such information can allow the instructor to determine ifthere are specific types of content items that a large amount ofindividuals are having difficulty with. If all those individualsparticipate in real-world instruction in a common setting, theinstructor can provide real-world instruction to the entire group ofindividuals and focus on those aspects of literacy that a large portionof the group are struggling with.

At box 2880, the computing system creates a collection of flash cards.As an illustration, a user with access to the user interface presentedin FIG. 29C may select, for any given row, the “Print” interface elementlisted under column 22. In response to selection of one of theseinterface elements, the computing system may generate a sheet of flashcards that list the content items from columns 20 and 21 for theappropriate participant. An example of such flash cards are those shownin FIG. 29B. An individual may then print off these flash cards so thatthe appropriate participant can practice those content items with whichthe participant struggled, and/or the user may send the flash cardselectronically to another individual to print off or review (e.g., aninstructor may send the flash cards to a parent of a student).

Overview of Example Computing Systems

Referring now to FIG. 31 , a conceptual diagram of a system that may beused to implement the systems and methods described in this document isillustrated. In the system, mobile computing device 3110 can wirelesslycommunicate with base station 3140, which can provide the mobilecomputing device wireless access to numerous hosted services 3160through a network 3150.

In this illustration, the mobile computing device 3110 is depicted as ahandheld mobile telephone (e.g., a smartphone, or an applicationtelephone) that includes a touchscreen display device 3112 forpresenting content to a user of the mobile computing device 3110 andreceiving touch-based user inputs and/or presence-sensitive user input(e.g., as detected over a surface of the computing device using radardetectors mounted in the mobile computing device 510). Other visual,tactile, and auditory output components may also be provided (e.g., LEDlights, a vibrating mechanism for tactile output, or a speaker forproviding tonal, voice-generated, or recorded output), as may variousdifferent input components (e.g., keyboard 3114, physical buttons,trackballs, accelerometers, gyroscopes, and magnetometers).

Example visual output mechanism in the form of display device 3112 maytake the form of a display with resistive or capacitive touchcapabilities. The display device may be for displaying video, graphics,images, and text, and for coordinating user touch input locations withthe location of displayed information so that the device 3110 canassociate user contact at a location of a displayed item with the item.The mobile computing device 3110 may also take alternative forms,including as a laptop computer, a tablet or slate computer, a personaldigital assistant, an embedded system (e.g., a car navigation system), adesktop personal computer, or a computerized workstation.

An example mechanism for receiving user-input includes keyboard 3114,which may be a full qwerty keyboard or a traditional keypad thatincludes keys for the digits ‘0-9’, ‘*’, and ‘#.’ The keyboard 3114receives input when a user physically contacts or depresses a keyboardkey. User manipulation of a trackball 3116 or interaction with a trackpad enables the user to supply directional and rate of movementinformation to the mobile computing device 3110 (e.g., to manipulate aposition of a cursor on the display device 3112).

The mobile computing device 3110 may be able to determine a position ofphysical contact with the touchscreen display device 3112 (e.g., aposition of contact by a finger or a stylus). Using the touchscreen3112, various “virtual” input mechanisms may be produced, where a userinteracts with a graphical user interface element depicted on thetouchscreen 3112 by contacting the graphical user interface element. Anexample of a “virtual” input mechanism is a “software keyboard,” where akeyboard is displayed on the touchscreen and a user selects keys bypressing a region of the touchscreen 3112 that corresponds to each key.

The mobile computing device 3110 may include mechanical or touchsensitive buttons 3118 a-d. Additionally, the mobile computing devicemay include buttons for adjusting volume output by the one or morespeakers 3120, and a button for turning the mobile computing device onor off. A microphone 3122 allows the mobile computing device 3110 toconvert audible sounds into an electrical signal that may be digitallyencoded and stored in computer-readable memory, or transmitted toanother computing device. The mobile computing device 3110 may alsoinclude a digital compass, an accelerometer, proximity sensors, andambient light sensors.

An operating system may provide an interface between the mobilecomputing device's hardware (e.g., the input/output mechanisms and aprocessor executing instructions retrieved from computer-readablemedium) and software. Example operating systems include ANDROID, CHROME,IOS, MAC OS X, WINDOWS 7, WINDOWS PHONE 7, SYMBIAN, BLACKBERRY, WEBOS, avariety of UNIX operating systems; or a proprietary operating system forcomputerized devices. The operating system may provide a platform forthe execution of application programs that facilitate interactionbetween the computing device and a user.

The mobile computing device 3110 may present a graphical user interfacewith the touchscreen 3112. A graphical user interface is a collection ofone or more graphical interface elements and may be static (e.g., thedisplay appears to remain the same over a period of time), or may bedynamic (e.g., the graphical user interface includes graphical interfaceelements that animate without user input).

A graphical interface element may be text, lines, shapes, images, orcombinations thereof. For example, a graphical interface element may bean icon that is displayed on the desktop and the icon's associated text.In some examples, a graphical interface element is selectable withuser-input. For example, a user may select a graphical interface elementby pressing a region of the touchscreen that corresponds to a display ofthe graphical interface element. In some examples, the user maymanipulate a trackball to highlight a single graphical interface elementas having focus. User-selection of a graphical interface element mayinvoke a pre-defined action by the mobile computing device. In someexamples, selectable graphical interface elements further oralternatively correspond to a button on the keyboard 3104.User-selection of the button may invoke the pre-defined action.

In some examples, the operating system provides a “desktop” graphicaluser interface that is displayed after turning on the mobile computingdevice 3110, after activating the mobile computing device 3110 from asleep state, after “unlocking” the mobile computing device 3110, orafter receiving user-selection of the “home” button 3118 c. The desktopgraphical user interface may display several graphical interfaceelements that, when selected, invoke corresponding application programs.An invoked application program may present a graphical interface thatreplaces the desktop graphical user interface until the applicationprogram terminates or is hidden from view.

User-input may influence an executing sequence of mobile computingdevice 3110 operations. For example, a single-action user input (e.g., asingle tap of the touchscreen, swipe across the touchscreen, contactwith a button, or combination of these occurring at a same time) mayinvoke an operation that changes a display of the user interface.Without the user-input, the user interface may not have changed at aparticular time. For example, a multi-touch user input with thetouchscreen 3112 may invoke a mapping application to “zoom-in” on alocation, even though the mapping application may have by defaultzoomed-in after several seconds.

The desktop graphical interface can also display “widgets.” A widget isone or more graphical interface elements that are associated with anapplication program that is executing, and that display on the desktopcontent controlled by the executing application program. A widget'sapplication program may launch as the mobile device turns on. Further, awidget may not take focus of the full display. Instead, a widget mayonly “own” a small portion of the desktop, displaying content andreceiving touchscreen user-input within the portion of the desktop.

The mobile computing device 3110 may include one or morelocation-identification mechanisms. A location-identification mechanismmay include a collection of hardware and software that provides theoperating system and application programs an estimate of the mobiledevice's geographical position. A location-identification mechanism mayemploy satellite-based positioning techniques, base station transmittingantenna identification, multiple base station triangulation, internetaccess point IP location determinations, inferential identification of auser's position based on search engine queries, and user-suppliedidentification of location (e.g., by receiving user a “check in” to alocation).

The mobile computing device 3110 may include other applications,computing sub-systems, and hardware. A call handling unit may receive anindication of an incoming telephone call and provide a user thecapability to answer the incoming telephone call. A media player mayallow a user to listen to music or play movies that are stored in localmemory of the mobile computing device 3110. The mobile device 3110 mayinclude a digital camera sensor, and corresponding image and videocapture and editing software. An internet browser may enable the user toview content from a web page by typing in an addresses corresponding tothe web page or selecting a link to the web page.

The mobile computing device 3110 may include an antenna to wirelesslycommunicate information with the base station 3140. The base station3140 may be one of many base stations in a collection of base stations(e.g., a mobile telephone cellular network) that enables the mobilecomputing device 3110 to maintain communication with a network 3150 asthe mobile computing device is geographically moved. The computingdevice 3110 may alternatively or additionally communicate with thenetwork 3150 through a Wi-Fi router or a wired connection (e.g.,ETHERNET, USB, or FIREWIRE). The computing device 3110 may alsowirelessly communicate with other computing devices using BLUETOOTHprotocols, or may employ an ad-hoc wireless network.

A service provider that operates the network of base stations mayconnect the mobile computing device 3110 to the network 3150 to enablecommunication between the mobile computing device 3110 and othercomputing systems that provide services 3160. Although the services 3160may be provided over different networks (e.g., the service provider'sinternal network, the Public Switched Telephone Network, and theInternet), network 3150 is illustrated as a single network. The serviceprovider may operate a server system 3152 that routes informationpackets and voice data between the mobile computing device 3110 andcomputing systems associated with the services 3160.

The network 3150 may connect the mobile computing device 3110 to thePublic Switched Telephone Network (PSTN) 3162 in order to establishvoice or fax communication between the mobile computing device 3110 andanother computing device. For example, the service provider serversystem 3152 may receive an indication from the PSTN 3162 of an incomingcall for the mobile computing device 3110. Conversely, the mobilecomputing device 3110 may send a communication to the service providerserver system 3152 initiating a telephone call using a telephone numberthat is associated with a device accessible through the PSTN 3162.

The network 3150 may connect the mobile computing device 3110 with aVoice over Internet Protocol (VoIP) service 3164 that routes voicecommunications over an IP network, as opposed to the PSTN. For example,a user of the mobile computing device 3110 may invoke a VoIP applicationand initiate a call using the program. The service provider serversystem 3152 may forward voice data from the call to a VoIP service,which may route the call over the internet to a corresponding computingdevice, potentially using the PSTN for a final leg of the connection.

An application store 3166 may provide a user of the mobile computingdevice 3110 the ability to browse a list of remotely stored applicationprograms that the user may download over the network 3150 and install onthe mobile computing device 3110. The application store 3166 may serveas a repository of applications developed by third-party applicationdevelopers. An application program that is installed on the mobilecomputing device 3110 may be able to communicate over the network 3150with server systems that are designated for the application program. Forexample, a VoIP application program may be downloaded from theApplication Store 3166, enabling the user to communicate with the VoIPservice 3164.

The mobile computing device 3110 may access content on the internet 3168through network 3150. For example, a user of the mobile computing device3110 may invoke a web browser application that requests data from remotecomputing devices that are accessible at designated universal resourcelocations. In various examples, some of the services 3160 are accessibleover the internet.

The mobile computing device may communicate with a personal computer3170. For example, the personal computer 3170 may be the home computerfor a user of the mobile computing device 3110. Thus, the user may beable to stream media from his personal computer 3170. The user may alsoview the file structure of his personal computer 3170, and transmitselected documents between the computerized devices.

A voice recognition service 3172 may receive voice communication datarecorded with the mobile computing device's microphone 3122, andtranslate the voice communication into corresponding textual data. Insome examples, the translated text is provided to a search engine as aweb query, and responsive search engine search results are transmittedto the mobile computing device 3110.

The mobile computing device 3110 may communicate with a social network3174. The social network may include numerous members, some of whichhave agreed to be related as acquaintances. Application programs on themobile computing device 3110 may access the social network 3174 toretrieve information based on the acquaintances of the user of themobile computing device. For example, an “address book” applicationprogram may retrieve telephone numbers for the user's acquaintances. Invarious examples, content may be delivered to the mobile computingdevice 3110 based on social network distances from the user to othermembers in a social network graph of members and connectingrelationships. For example, advertisement and news article content maybe selected for the user based on a level of interaction with suchcontent by members that are “close” to the user (e.g., members that are“friends” or “friends of friends”).

The mobile computing device 3110 may access a personal set of contacts3176 through network 3150. Each contact may identify an individual andinclude information about that individual (e.g., a phone number, anemail address, and a birthday). Because the set of contacts is hostedremotely to the mobile computing device 3110, the user may access andmaintain the contacts 3176 across several devices as a common set ofcontacts.

The mobile computing device 3110 may access cloud-based applicationprograms 3178. Cloud-computing provides application programs (e.g., aword processor or an email program) that are hosted remotely from themobile computing device 3110, and may be accessed by the device 3110using a web browser or a dedicated program. Example cloud-basedapplication programs include GOOGLE DOCS word processor and spreadsheetservice, GOOGLE GMAIL webmail service, and PICASA picture manager.

Mapping service 3180 can provide the mobile computing device 3110 withstreet maps, route planning information, and satellite images. Anexample mapping service is GOOGLE MAPS. The mapping service 3180 mayalso receive queries and return location-specific results. For example,the mobile computing device 3110 may send an estimated location of themobile computing device and a user-entered query for “pizza places” tothe mapping service 3180. The mapping service 3180 may return a streetmap with “markers” superimposed on the map that identify geographicallocations of nearby “pizza places.”

Turn-by-turn service 3182 may provide the mobile computing device 3110with turn-by-turn directions to a user-supplied destination. Forexample, the turn-by-turn service 3182 may stream to device 3110 astreet-level view of an estimated location of the device, along withdata for providing audio commands and superimposing arrows that direct auser of the device 3110 to the destination.

Various forms of streaming media 3184 may be requested by the mobilecomputing device 3110. For example, computing device 3110 may request astream for a pre-recorded video file, a live television program, or alive radio program. Example services that provide streaming mediainclude YOUTUBE and PANDORA.

A micro-blogging service 3186 may receive from the mobile computingdevice 3110 a user-input post that does not identify recipients of thepost. The micro-blogging service 3186 may disseminate the post to othermembers of the micro-blogging service 3186 that agreed to subscribe tothe user.

A search engine 3188 may receive user-entered textual or verbal queriesfrom the mobile computing device 3110, determine a set ofinternet-accessible documents that are responsive to the query, andprovide to the device 3110 information to display a list of searchresults for the responsive documents. In examples where a verbal queryis received, the voice recognition service 3172 may translate thereceived audio into a textual query that is sent to the search engine.

These and other services may be implemented in a server system 3190. Aserver system may be a combination of hardware and software thatprovides a service or a set of services. For example, a set ofphysically separate and networked computerized devices may operatetogether as a logical server system unit to handle the operationsnecessary to offer a service to hundreds of computing devices. A serversystem is also referred to herein as a computing system.

In various implementations, operations that are performed “in responseto” or “as a consequence of” another operation (e.g., a determination oran identification) are not performed if the prior operation isunsuccessful (e.g., if the determination was not performed). Operationsthat are performed “automatically” are operations that are performedwithout user intervention (e.g., intervening user input). Features inthis document that are described with conditional language may describeimplementations that are optional. In some examples, “transmitting” froma first device to a second device includes the first device placing datainto a network for receipt by the second device, but may not include thesecond device receiving the data. Conversely, “receiving” from a firstdevice may include receiving the data from a network, but may notinclude the first device transmitting the data.

“Determining” by a computing system can include the computing systemrequesting that another device perform the determination and supply theresults to the computing system. Moreover, “displaying” or “presenting”by a computing system can include the computing system sending data forcausing another device to display or present the referenced information.

FIG. 32 is a block diagram of computing devices 3200, 3250 that may beused to implement the systems and methods described in this document, aseither a client or as a server or plurality of servers. Computing device3200 is intended to represent various forms of digital computers, suchas laptops, desktops, workstations, personal digital assistants,servers, blade servers, mainframes, and other appropriate computers.Computing device 3250 is intended to represent various forms of mobiledevices, such as personal digital assistants, cellular telephones,smartphones, and other similar computing devices. The components shownhere, their connections and relationships, and their functions, aremeant to be examples only, and are not meant to limit implementationsdescribed and/or claimed in this document.

Computing device 3200 includes a processor 3202, memory 3204, a storagedevice 3206, a high-speed interface 3208 connecting to memory 3204 andhigh-speed expansion ports 3210, and a low speed interface 3212connecting to low speed bus 3214 and storage device 3206. Each of thecomponents 3202, 3204, 3206, 3208, 3210, and 3212, are interconnectedusing various busses, and may be mounted on a common motherboard or inother manners as appropriate. The processor 3202 can processinstructions for execution within the computing device 3200, includinginstructions stored in the memory 3204 or on the storage device 3206 todisplay graphical information for a GUI on an external input/outputdevice, such as display 3216 coupled to high-speed interface 3208. Inother implementations, multiple processors and/or multiple buses may beused, as appropriate, along with multiple memories and types of memory.Also, multiple computing devices 3200 may be connected, with each deviceproviding portions of the necessary operations (e.g., as a server bank,a group of blade servers, or a multi-processor system).

The memory 3204 stores information within the computing device 3200. Inone implementation, the memory 3204 is a volatile memory unit or units.In another implementation, the memory 3204 is a non-volatile memory unitor units. The memory 3204 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 3206 is capable of providing mass storage for thecomputing device 3200. In one implementation, the storage device 3206may be or contain a computer-readable medium, such as a floppy diskdevice, a hard disk device, an optical disk device, or a tape device, aflash memory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. A computer program product can be tangibly embodied inan information carrier. The computer program product may also containinstructions that, when executed, perform one or more methods, such asthose described above. The information carrier is a computer- ormachine-readable medium, such as the memory 3204, the storage device3206, or memory on processor 3202.

The high-speed controller 3208 manages bandwidth-intensive operationsfor the computing device 3200, while the low speed controller 3212manages lower bandwidth-intensive operations. Such allocation offunctions is an example only. In one implementation, the high-speedcontroller 3208 is coupled to memory 3204, display 3216 (e.g., through agraphics processor or accelerator), and to high-speed expansion ports3210, which may accept various expansion cards (not shown). In theimplementation, low-speed controller 3212 is coupled to storage device3206 and low-speed expansion port 3214. The low-speed expansion port,which may include various communication ports (e.g., USB, Bluetooth,Ethernet, wireless Ethernet) may be coupled to one or more input/outputdevices, such as a keyboard, a pointing device, a scanner, or anetworking device such as a switch or router, e.g., through a networkadapter.

The computing device 3200 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 3220, or multiple times in a group of such servers. Itmay also be implemented as part of a rack server system 3224. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 3222. Alternatively, components from computing device 3200 maybe combined with other components in a mobile device (not shown), suchas device 3250. Each of such devices may contain one or more ofcomputing device 3200, 3250, and an entire system may be made up ofmultiple computing devices 3200, 3250 communicating with each other.

Computing device 3250 includes a processor 3252, memory 3264, aninput/output device such as a display 3254, a communication interface3266, and a transceiver 3268, among other components. The device 3250may also be provided with a storage device, such as a microdrive orother device, to provide additional storage. Each of the components3250, 3252, 3264, 3254, 3266, and 3268, are interconnected using variousbuses, and several of the components may be mounted on a commonmotherboard or in other manners as appropriate.

The processor 3252 can execute instructions within the computing device3250, including instructions stored in the memory 3264. The processormay be implemented as a chipset of chips that include separate andmultiple analog and digital processors. Additionally, the processor maybe implemented using any of a number of architectures. For example, theprocessor may be a CISC (Complex Instruction Set Computers) processor, aRISC (Reduced Instruction Set Computer) processor, or a MISC (MinimalInstruction Set Computer) processor. The processor may provide, forexample, for coordination of the other components of the device 3250,such as control of user interfaces, applications run by device 3250, andwireless communication by device 3250.

Processor 3252 may communicate with a user through control interface3258 and display interface 3256 coupled to a display 3254. The display3254 may be, for example, a TFT (Thin-Film-Transistor Liquid CrystalDisplay) display or an OLED (Organic Light Emitting Diode) display, orother appropriate display technology. The display interface 3256 maycomprise appropriate circuitry for driving the display 3254 to presentgraphical and other information to a user. The control interface 3258may receive commands from a user and convert them for submission to theprocessor 3252. In addition, an external interface 3262 may be providein communication with processor 3252, so as to enable near areacommunication of device 3250 with other devices. External interface 3262may provided, for example, for wired communication in someimplementations, or for wireless communication in other implementations,and multiple interfaces may also be used.

The memory 3264 stores information within the computing device 3250. Thememory 3264 can be implemented as one or more of a computer-readablemedium or media, a volatile memory unit or units, or a non-volatilememory unit or units. Expansion memory 3274 may also be provided andconnected to device 3250 through expansion interface 3272, which mayinclude, for example, a SIMM (Single In Line Memory Module) cardinterface. Such expansion memory 3274 may provide extra storage spacefor device 3250, or may also store applications or other information fordevice 3250. Specifically, expansion memory 3274 may includeinstructions to carry out or supplement the processes described above,and may include secure information also. Thus, for example, expansionmemory 3274 may be provide as a security module for device 3250, and maybe programmed with instructions that permit secure use of device 3250.In addition, secure applications may be provided via the SIMM cards,along with additional information, such as placing identifyinginformation on the SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory,as discussed below. In one implementation, a computer program product istangibly embodied in an information carrier. The computer programproduct contains instructions that, when executed, perform one or moremethods, such as those described above. The information carrier is acomputer- or machine-readable medium, such as the memory 3264, expansionmemory 3274, or memory on processor 3252 that may be received, forexample, over transceiver 3268 or external interface 3262.

Device 3250 may communicate wirelessly through communication interface3266, which may include digital signal processing circuitry wherenecessary. Communication interface 3266 may provide for communicationsunder various modes or protocols, such as GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others.Such communication may occur, for example, through radio-frequencytransceiver 3268. In addition, short-range communication may occur, suchas using a Bluetooth, WiFi, or other such transceiver (not shown). Inaddition, GPS (Global Positioning System) receiver module 3270 mayprovide additional navigation- and location-related wireless data todevice 3250, which may be used as appropriate by applications running ondevice 3250.

Device 3250 may also communicate audibly using audio codec 3260, whichmay receive spoken information from a user and convert it to usabledigital information. Audio codec 3260 may likewise generate audiblesound for a user, such as through a speaker, e.g., in a handset ofdevice 3250. Such sound may include sound from voice telephone calls,may include recorded sound (e.g., voice messages, music files, etc.) andmay also include sound generated by applications operating on device3250.

The computing device 3250 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as acellular telephone 3280. It may also be implemented as part of asmartphone 3282, personal digital assistant, or other similar mobiledevice.

Additionally computing device 3200 or 3250 can include Universal SerialBus (USB) flash drives. The USB flash drives may store operating systemsand other applications. The USB flash drives can include input/outputcomponents, such as a wireless transmitter or USB connector that may beinserted into a USB port of another computing device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium”“computer-readable medium” refers to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), peer-to-peernetworks (having ad-hoc or static members), grid computinginfrastructures, and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

Although a few implementations have been described in detail above,other modifications are possible. Moreover, other mechanisms forperforming the systems and methods described in this document may beused. In addition, the logic flows depicted in the figures do notrequire the particular order shown, or sequential order, to achievedesirable results. Other steps may be provided, or steps may beeliminated, from the described flows, and other components may be addedto, or removed from, the described systems. Accordingly, otherimplementations are within the scope of the following claims.

1-20. (canceled)
 21. A computer-implemented method for analyzing amanner in which touchscreen user inputs are generated, comprising:presenting, by a computing system, a multi-character word and promptinga user of the computing system to write the multi-character word usingan input device of the computing system; receiving, by the computingsystem, user input that interacts with the input device of the computingsystem to generate a user-animated gesture, responsive to the computingsystem prompting the user to write the multi-character word using theinput device, wherein a final form of the user-animated gesturesatisfactorily represents the multi-character word; identifying, by thecomputing system, different portions of the user-animated gesture thatcorrespond to different respective characters of the multi-characterword; for each respective character in the multi-character word: (i)accessing, by the computing system, a set of pre-stored animatedgestures that represent different techniques for forming the respectivecharacter; (ii) determining, by the computing system, a first level ofsimilarity between the portion of the user-animated gesture thatcorresponds to the respective character and a first pre-stored animatedgesture for forming the respective character from among the set ofpre-stored animated gestures for forming the respective character,wherein the first pre-stored animated gesture for forming the respectivecharacter is designated as representing a satisfactory technique inwhich to form the respective character; (iii) determining, by thecomputing system, a second level of similarity between the portion ofthe user-animated gesture that corresponds to the respective characterand a second pre-stored animated gesture for forming the respectivecharacter from among the set of pre-stored animated gestures for formingthe respective character, wherein the second pre-stored animated gesturefor forming the respective character is designated as representing anunsatisfactory technique in which to form the respective character; (iv)identifying, by the computing system, which of the first pre-storedanimated gesture and the second pre-stored animated gesture matches theportion of the user-animated gesture that corresponds to the respectivecharacter based on which of the first level of similarity and the secondlevel of similarity represents a greatest level of similarity; (v)storing, by the computing system, an indication regarding whether theuser-animated gesture formed the respective character using asatisfactory technique, based on which of the first pre-stored animatedgesture and the second pre-stored animated gesture the computing systemdetermined to match the user-animated gesture, wherein the computingsystem identifies that the user-animated gesture formed a firstcharacter of the multi-character word using a satisfactory technique andidentifies that the user-animated gesture formed a second character ofthe multi-character word using an unsatisfactory technique; anddisplaying, by the computing system, an indication that theuser-animated gesture formed the second character using anunsatisfactory technique, based on the computing system determining thata portion of the user-animated gesture that corresponds to the secondcharacter has a greatest level of similarity with a pre-stored animatedgesture that represents an unsatisfactory technique for forming thesecond character rather than a pre-stored animated gesture thatrepresents a satisfactory technique for forming the second character.22. The computer-implemented method of claim 21, wherein presenting themulti-character word includes displaying a textual representation of themulti-character word on a touchscreen.
 23. The computer-implementedmethod of claim 21, wherein identifying which of the first pre-storedanimated gesture and the second pre-stored animated gesture matches theuser-animated gesture based on which of the first level of similarityand the second level of similarity represents a greatest level ofsimilarity includes determining that one of the first level ofsimilarity or the second level of similarity satisfies a designatedthreshold level of similarity.
 24. The computer-implemented method ofclaim 21, wherein: the first pre-stored animated gesture for forming therespective character represents a technique in which the respectivecharacter is formed by starting at a first location of the respectivecharacter; and the second pre-stored animated gesture for forming therespective character represents a technique in which the respectivecharacter is formed by starting formation of the respective characterstarting at a second location of the respective character, the secondlocation being on a different side of the respective character than thefirst location.
 25. The computer-implemented method of claim 21,wherein: the method further comprises, for each respective character inthe multi-character word, determining, by the computing system, a thirdlevel of similarity between the portion of the user-animated gesturethat corresponds to the respective character and a third pre-storedanimated gesture for forming the respective character from among the setof pre-stored animated gestures for forming the respective character;the second pre-stored animated gesture for forming the respectivecharacter represents a first type of unsatisfactory technique in whichto form the respective character; the third pre-stored animated gesturefor forming the respective character represents a second type ofunsatisfactory technique in which to form the respective character; thecomputing system identifying that the user-animated gesture formed thesecond character of the multi-character word using an unsatisfactorytechnique includes the computing system identifying that theuser-animated gesture formed the second character using the first typeof unsatisfactory technique; the computing system is configured toidentify that the user-animated gesture formed the second characterusing the second type of unsatisfactory technique responsive to thecomputing system determining that the portion of the user-animatedgesture that corresponds to the second character has a greatest level ofsimilarity with the third type of pre-stored animated gesture accessedfor the second character.
 26. The computer-implemented method of claim25, wherein the second type of unsatisfactory technique represented bythe third pre-stored animated gesture for forming the respectivecharacter includes the respective character being formed backwards. 27.The computer-implemented method of claim 21, wherein: determining, bythe computing system, the first level of similarity includes:coordinating, by the computing system, a timing of the portion of theuser-animated gesture that corresponds to the respective character and afirst timing of the first pre-stored animated gesture for forming therespective character; determining, for each respective time of multiplefirst coordinated times after the timing and the first timing have beencoordinated, a first distance between (a) a location of user inputforming the user-animated gesture at the respective time, and (b) alocation representing user input forming the first pre-stored animatedgesture at the respective time; and generating, by the computing system,the first level of similarity based on a combination of the firstdistances determined for the multiple first coordinated times; anddetermining, by the computing system, the second level of similarityincludes: coordinating, by the computing system, the timing of theportion of the user-animated gesture that corresponds to the respectivecharacter and a second timing of the second pre-stored animated gesturefor forming the respective character; determining, for each respectivetime of multiple second coordinated times after the timing and thesecond timing have been coordinated, a second distance between (a) alocation of user input forming the user-animated gesture at therespective time, and (b) a location representing user input forming thesecond pre-stored animated gesture at the respective time; andgenerating, by the computing system, the second level of similaritybased on a combination of the second distances determined for themultiple second coordinated times.
 28. The computer-implemented methodof claim 21, further comprising: presenting, by the computing system, asecond multi-character word and prompting the user of the computingsystem to write the second multi-character word using the input deviceof the computing system; receiving, by the computing system, user inputthat interacts with the input device to generate a second user-animatedgesture, responsive to the computing system prompting the user to writethe second multi-character word with the input device; identifying, bythe computing system, different portions of the second user-animatedgesture that correspond to different respective characters of the secondmulti-character word; determining, by the computing system through acomparison of the second user-animated gesture to the secondmulti-character word, that the second user-animated gesture does notmatch the second multi-character word, and instead matches the secondmulti-character word with one of (a) an extra letter inserted, (b) aletter missing, and (c) letters ordered incorrectly; displaying, by thecomputing system, an indication that the second user-animated gestureformed the second multi-character word with one of (a) an extra letterinserted, (b) a letter missing, and (c) letters ordered incorrectly,based on the comparison of the second user-animated gesture to thesecond multi-character word.
 29. The computer-implemented method ofclaim 21, wherein: the computing system includes a server system, andthe server system performs (i) the determining the first level ofsimilarity between the portion of the user-animated gesture and thefirst pre-stored animated gesture, and (ii) the determining the secondlevel of similarity between the portion of the user-animated gesture andthe second pre-stored animated gesture; the presenting themulti-character word involves the server system providing information toa handheld computing device to cause the handheld computing device topresent the multi-character word; the receiving the user input involvesthe server system receiving from the handheld computing device anindication that the user input was provided at the handheld computingdevice; and the displaying the indication that the user-animated gestureformed the second character using an unsatisfactory technique involvesthe server system providing information to a second computing device tocause the second computing device to display text that specifies thatthe user of the computing system formed the second character using anunsatisfactory technique.
 30. A computing system that comprises: one ormore processors; and one or more computer-readable devices includinginstructions that, when executed by the one or more processors, causethe computing system to perform operations that include: presenting, bythe computing system, a multi-character word and prompting a user of thecomputing system to write the multi-character word using an input deviceof the computing system; receiving, by the computing system, user inputthat interacts with the input device of the computing system to generatea user-animated gesture, responsive to the computing system promptingthe user to write the multi-character word using the input device,wherein a final form of the user-animated gesture satisfactorilyrepresents the multi-character word; identifying, by the computingsystem, different portions of the user-animated gesture that correspondto different respective characters of the multi-character word; for eachrespective character in the multi-character word: (i) accessing, by thecomputing system, a set of pre-stored animated gestures that representdifferent techniques for forming the respective character; (ii)determining, by the computing system, a first level of similaritybetween the portion of the user-animated gesture that corresponds to therespective character and a first pre-stored animated gesture for formingthe respective character from among the set of pre-stored animatedgestures for forming the respective character, wherein the firstpre-stored animated gesture for forming the respective character isdesignated as representing a satisfactory technique in which to form therespective character; (iii) determining, by the computing system, asecond level of similarity between the portion of the user-animatedgesture that corresponds to the respective character and a secondpre-stored animated gesture for forming the respective character fromamong the set of pre-stored animated gestures for forming the respectivecharacter, wherein the second pre-stored animated gesture for formingthe respective character is designated as representing an unsatisfactorytechnique in which to form the respective character; (iv) identifying,by the computing system, which of the first pre-stored animated gestureand the second pre-stored animated gesture matches the portion of theuser-animated gesture that corresponds to the respective character basedon which of the first level of similarity and the second level ofsimilarity represents a greatest level of similarity; (v) storing, bythe computing system, an indication regarding whether the user-animatedgesture formed the respective character using a satisfactory technique,based on which of the first pre-stored animated gesture and the secondpre-stored animated gesture the computing system determined to match theuser-animated gesture, wherein the computing system identifies that theuser-animated gesture formed a first character of the multi-characterword using a satisfactory technique and identifies that theuser-animated gesture formed a second character of the multi-characterword using an unsatisfactory technique; and displaying, by the computingsystem, an indication that the user-animated gesture formed the secondcharacter using an unsatisfactory technique, based on the computingsystem determining that a portion of the user-animated gesture thatcorresponds to the second character has a greatest level of similaritywith a pre-stored animated gesture that represents an unsatisfactorytechnique for forming the second character rather than a pre-storedanimated gesture that represents a satisfactory technique for formingthe second character.